Electronic employee selection systems and methods

ABSTRACT

An automated employee selection system can use a variety of techniques to provide information for assisting in selection of employees. For example, pre-hire and post-hire information can be collected electronically and used to build an artificial-intelligence based model. The model can then be used to predict a desired job performance criterion (e.g., tenure, number of accidents, sales level, or the like) for new applicants. A wide variety of features can be supported, such as electronic reporting. Pre-hire information identified as ineffective can be removed from a collected pre-hire information. For example, ineffective questions can be identified and removed from a job application. New items can be added and their effectiveness tested. As a result, a system can exhibit adaptive learning and maintain or increase effectiveness even under changing conditions.

RELATED APPLICATION DATA

[0001] This application claims the benefit of Becker et al., U.S.Provisional Patent Application No. 60/223,289, “Automated electronicemployee selection process and hiring recommendation system usingon-line criterion validation data collection, artificialintelligence-enabled adaptive learning and electronic resultsreporting,” filed Aug. 3, 2000.

TECHNICAL FIELD

[0002] The invention relates to automated employee selection.

COPYRIGHT AUTHORIZATION

[0003] A portion of the disclosure of this patent document containsmaterial that is subject to copyright protection. The copyright ownerhas no objection to the facsimile reproduction by anyone of the patentdocument or the patent disclosure, as it appears in the Patent andTrademark Office patent file or records, but otherwise reserves allcopyright rights whatsoever.

BACKGROUND

[0004] Organizations can spend considerable time and effort identifyingand hiring suitable employees. Good help is hard to find. Despite theirbest efforts, organizations still often meet with failure and simplyaccept high turnover and poor employee performance.

[0005] A variety of approaches to finding and hiring employees have beentried. A well-known tool for employee selection is the job application.Job applications help identify a job applicant's qualifications, such aseducational background, job history, skills, and experience.

[0006] An employer typically collects a set of job applications fromapplicants who drop by an employer work site or appear at a job fair.Someone in the organization then reviews the applications to determinewhich applicants merit further investigation. Then, a job interview, atest, or some other review process is sometimes used to further limitthe applicant pool.

[0007] With the advent of the electronic age, job applications can becompleted electronically. In this way, the delays associated withprocessing paper can be minimized. However, evenelectronically-completed job applications can be of questionable meritand still require considerable effort on the part of the hiringorganization to review them. A better way of selecting employees isstill needed.

SUMMARY

[0008] Large organizations can bring considerable resources to bear onthe task of developing a job application. For example, a large retailchain might consult with an industrial psychologist to study the jobenvironment and develop a set of questions that ostensibly predictwhether an individual will excel in the environment.

[0009] However, such an approach is fraught with inaccuracy andsubjectivity; further, the psychologist's analysis depends on conditionsthat may change over time. For example, even if the psychologistidentifies appropriate factors for testing, an applicant might slantanswers on the application based on what the applicant perceives isexpected. Further, two psychologists might come up with two completelydifferent sets of factors. And, finally, as the job conditions andapplicant pool changes over time, the factors may become less effectiveor ineffective.

[0010] To determine whether a job application is effective, a study canbe conducted to verify whether the factors chosen by the psychologisthave been successful in identifying suitable applicants. However, such astudy requires even more effort in addition to the considerable effortalready invested in developing the application. So, such a studytypically is not conducted until managers in the organization alreadyknow that the application is ineffective or out of date.

[0011] The disclosed embodiments include various systems and methodsrelated to automated employee selection. For example, various techniquescan be used to automate the job application and employee selectionprocess.

[0012] In one aspect of an embodiment, answers to job applicationquestions can be collected directly from the applicant via an electronicdevice. Based on correlations of the answers with answers to questionsby other individuals for which post-hire information has been collected,a post-hire outcome is predicted.

[0013] In another aspect of an embodiment, an artificial-intelligencetechnique is used. For example, a neural network or a fuzzy logic systemcan be used to build a model that predicts a post-hire outcome. Proposedmodels of different types can be constructed and tested to identify asuperior model.

[0014] When constructing a model, an information-theory-based featureselection technique can be used to reduce the number of inputs, therebyfacilitating more efficient model construction.

[0015] Items identified as ineffective predictors can be removed fromthe job application. Information collected based on the new jobapplication can be used to build a refined model. In this way, a systemcan exhibit adaptive learning and maintain its effectiveness even ifconditions change over time. Content can be rotated or otherwisemodified so the job application changes and maintains its effectivenessover time. Evolution toward higher predictive accuracy for employeeselection can be achieved.

[0016] A sample size monitor can identify when sufficient informationhas been collected electronically to build a refined model. In this way,short-cycle criterion validation and performance-driven item rotationcan be supported.

[0017] Outcomes can be predicted for any of a wide variety of parametersand be provided in various formats. For example, tenure, number ofaccidents, sales level, whether the employee will be involuntarilyterminated, whether the employee will be eligible for rehire upontermination and other measures of employee effectiveness can bepredicted. The prediction can be provided in a variety of forms, suchas, for example, in the form of a predicted value, a predicted rank, apredicted range, or a predicted probability that an individual willbelong to a group.

[0018] Predictions can be provided by electronic means. For example,upon analysis of a job applicant's answers, an email or fax can be sentto a hiring manager indicating a favorable recommendation regarding theapplicant. In this way, real-time processing of a job application toprovide a recommendation can be supported.

[0019] Information from various predictors can be combined to provide aparticularly effective prediction. For example, a prediction can bebased at least on whether (or the likelihood) the applicant will beinvoluntarily terminated and whether (or the likelihood) the applicantwill be eligible for rehire upon termination. Based on whether theindividual is predicted to both voluntarily quit and be eligible forrehire upon termination, an accurate measure of the predictedsuitability of an applicant can be provided.

[0020] Post-hire information can be based on payroll information. Forexample, termination status and eligibility for rehire information canbe identified by examining payroll records. The payroll information canbe provided electronically to facilitate a high-level of accuratepost-hire information collection.

[0021] Further, reports can be provided to indicate a wide-variety ofparameters, such as applicant flow, effectiveness of the system, andothers.

[0022] Although the described technologies can continue to use theservices of an industrial psychologist, relationships between pre-hiredata predictors and desired job performance criteria can be discoveredand used without regard to whether the psychologist would predict such arelationship. A system using the described technologies can findrelationships in data that may elude a human researcher.

[0023] Additional features and advantages of the various embodimentswill be made apparent from the following detailed description ofillustrated embodiments, which proceeds with reference to theaccompanying drawings.

[0024] The present invention includes all novel and nonobvious features,method steps, and acts alone and in various combinations andsub-combinations with one another as set forth in the claims below. Thepresent invention is not limited to a particular combination orsub-combination.

BRIEF DESCRIPTION OF THE DRAWINGS

[0025]FIG. 1 is a block diagram showing exemplary pre-hire informationcollection.

[0026]FIG. 2 is a block diagram showing a predictive model based onpre-hire and post-hire information.

[0027]FIG. 3 is a block diagram showing ineffective predictors based onpre-hire and post-hire information.

[0028]FIG. 4 is a block diagram showing refinement of a model over time.

[0029]FIG. 5 is a flowchart showing a method for refining a model overtime.

[0030]FIG. 6 is a block diagram showing an exemplary system forproviding employee suitability recommendations.

[0031]FIG. 7 is a flowchart illustrating an exemplary method forproviding employee suitability recommendations.

[0032]FIG. 8 is a block diagram illustrating an exemplary architecturefor providing employee suitability recommendations.

[0033]FIG. 9 is a flowchart illustrating an exemplary method forbuilding a predictive model.

[0034]FIG. 10 is a block diagram showing an exemplary predictive model.

[0035]FIG. 11 is a block diagram showing an exemplary refined predictivemodel.

[0036]FIG. 12 is a block diagram illustrating integration of payrollinformation into a predictive system.

[0037]FIG. 13 is a block diagram illustrating an exemplary combinationof elements into a system.

[0038] FIGS. 14A-14D are block diagrams illustrating an exemplaryprocess for implementing automated employee selection.

[0039]FIG. 15 is a process flow diagram illustrating an exemplaryprocess for an employment suitability prediction system.

[0040]FIG. 16 is a graph illustrating exemplary effectiveness of asystem over time.

[0041]FIG. 17 is a graph illustrating entropy.

DETAILED DESCRIPTION Overview of the Technologies

[0042] On a general level, the described technologies can includecollecting information and building a model based on the information.Such a model can then be used to generate a prediction for one or moredesired job performance-related criteria. The prediction can be thebasis of a hiring recommendation or other employee selectioninformation.

[0043] Pre-hire information includes any information collected about anindividual before the individual (e.g., a job applicant or othercandidate) is hired. FIG. 1 shows a variety of sources 102 forcollecting pre-hire information 112. The pre-hire information 112 can bestored in electronic (e.g., digital) form in a computer-readable medium(e.g., RAM, ROM, magnetic disk, CD-ROM, CD-R, DVD-ROM, and the like).Possible sources for pre-hire information 112 include a paper-basedsource 122, an electronic device 124, a third party service 126, or someother source 128. For example, pre-hire information can include anapplicant's answers to an on-line employment application collected at aremote site, such as at an electronic device located in a kiosk at aprospective employer's work site. Further information and examples aredescribed in “Example 2—Collecting Information,” below.

[0044] Post-hire information includes any information collected about anindividual (e.g., an employee) after the individual is hired, includinginformation collected while the employee is employed or after anemployee is fired, laid off, or quits. Post-hire information cansimilarly be collected from a wide variety of sources. Post-hireinformation can include information about the employee's terminationdate. Further examples are described in “Example 2—CollectingInformation,” below.

[0045] As shown in FIG. 2, after pre-hire information 212 and post-hireinformation 222 have been collected, a predictive model 232 can bebuilt. As described in more detail below, a predictive model 232 cantake a variety of forms, including artificial intelligence-based models.The predictive model can generate one or more predictions based onpre-hire information inputs. Thus, the model can be used to generatepredictions for job applicants. In practice, the model can beimplemented as computer-executable code stored in a computer-readablemedium.

[0046] As shown in FIG. 3, after pre-hire information 312 and post-hireinformation 322 have been collected, ineffective predictors 332 can beidentified. Such ineffective predictors can be ignored when constructinga model (e.g., the model 232 of FIG. 2). In this way, the complexity ofthe model can be reduced, and the efficiency of the model constructionprocess can be improved.

[0047] Further, the same ineffective predictors 332 or similarineffective predictors can be removed from pre-hire content (e.g.,ineffective questions can be removed from a job application).Identification of ineffective predictors can be achieved via softwareusing a variety of techniques; examples are described below.

[0048] As shown in FIG. 4, using various features described herein, apredictive model M₁ (412) based on pre-hire information PR₁ (414) andpost-hire information PO₁ (416) can be refined. For example, informationcollection techniques can be refined by removing pre-hire contentidentified as ineffective. Further, additional pre-hire content might beadded (e.g., a new set of questions can be added to a job application).

[0049] As a result, new pre-hire information PR₂ (424) based on therefined pre-hire content can be collected. Corresponding post-hireinformation PO₂ (426) can be collected. Based on the information, arefined model M₂ (422) can be constructed.

[0050] The refinement process can be continued. For example, theeffectiveness of the additional pre-hire content can be determined.Thus, refinement can continue a number of times over time, resulting inpre-hire information PR_(n) (444), post-hire information PO_(n) (446),and a refined model M_(n) (442).

[0051]FIG. 5 shows an exemplary method for refining a predictive model.At 522, pre-hire information for applicants is collected based onpre-hire content (e.g., predictors such as questions on an employmentapplication or predictors collected from other sources). At 532,post-hire information for the applicants is collected. At 542 apredictive model is constructed. The model can be deployed and modeloutput used for hiring recommendations. At 552, the pre-hire content canbe refined (e.g., one or more ineffective questions can be removed andone or new ones can be added). Then, additional pre-hire information canbe collected at 522 (e.g., based on the refined pre-hire content).Eventually, a refined model can be generated.

[0052] The various models shown can be used as a basis for providingemployee hiring recommendations. The architecture used to implement anelectronic system providing such employee hiring recommendations canvary from simple to complex. FIG. 6 shows an overview of an exemplarysystem 602. In the example, a computer-based electronic device 612housed in a kiosk is situated in a work site (e.g., a retail store) andpresents a job application to a job applicant via an electronic display614. The electronic device then sends the applicant's answers to acentral server 622, which can also receive information from otherelectronic devices, such as the electronic device 624.

[0053] The server 622 can save the answers to a database 626 andimmediately apply a predictive model to the answers to generate one ormore predictions of employment performance for the applicant and ahiring recommendation based on the predictions. Thus, real-timeprocessing of incoming data can be accomplished.

[0054] The hiring recommendation can be immediately sent to a hiringmanager's computer 642 via a network 652 (e.g., in an email via theInternet). Thus, real-time reporting based on incoming data can beaccomplished. Although often less desirable, delayed processing is alsopossible. Thus, alternatively, the system can, for example, queueinformation and send it out in batches (e.g., in a set of n applicantsor every n days) as desired.

[0055] Various combinations and sub-combinations of the techniques belowcan be applied to any of the above examples.

EXAMPLE 1 Exemplary System and Method

[0056]FIG. 7 is a flowchart showing an exemplary method 702 forproviding automated employee selection. At 712, questions are asked ofan applicant such as via an electronic device. The answers are collectedat 722. Based on the answers, a prediction is generated at 732. Then,the results are provided at 742.

[0057]FIG. 8 is a block diagram an exemplary system 802 for providingemployee selection. An electronic data interrogator 812 is operable topresent a first set of a plurality of questions to an individual. Anelectronic answer capturer 822 is operable to electronically store theindividual's responses to at least a selected plurality of the first setof questions presented to the individual.

[0058] An electronic applicant predictor 832 is responsive to the storedanswers and is operable to predict at least one post-hire outcome if theindividual were to be employed by the employer. The applicant predictor832 can provide a prediction of the outcome based on correlations of thestored answers with answers to sets of the same questions by otherindividuals for which post-hire information has been collected. Thepredictor 832 can include a model constructed according to techniquesdescribed herein, such as in “Example 3—Building a Predictive Model” andothers.

[0059] An electronic results provider 842 can provide an outputindicating the outcome to assist in determining the suitability of theindividual for employment by an employer.

[0060] Some actions or elements might be performed or implemented bydifferent parties and are therefore not necessarily included in aparticular method or system. For example, collection of data might beperformed by one organization, and another might generate theprediction.

EXAMPLE 2 Collecting Information

[0061] As described with reference to FIG. 1 above, pre-hire informationcan be a variety of information collected from a variety of sources. Onepossible source for pre-hire information is a paper-based collectionsource 122, such as a paper-based job application or test. Paper-basedsources can be converted into electronic form by manual data entry orscanning.

[0062] Another possible source is an electronic device 124. Such anelectronic device can, for example, be a computer, a computer-basedkiosk, a screen phone, a telephone, or a biometric device. For example,pre-hire content (e.g., a job application or skills test) can bepresented to an applicant, who responds (e.g., answers questions)directly on the electronic device 124. Questions can be logicallyconnected so that they are presented only if appropriate (e.g., if theemployee answers affirmative to a question about termination, the devicecan then inquire as to the reason for termination).

[0063] Still another possible source for pre-hire information 112 isfrom a third party service 126. For example, credit reporting agencies,background check services, and other services can provide informationeither manually or over an online connection.

[0064] Yet another possible source for pre-hire information 112 is fromanother source 128. For example, later-developed technologies can beincorporated.

[0065] Any of the pre-hire information can be collected from a remotelocation (e.g., at a work site or from the applicant's home). Theinformation 112 can then be stored in a central location, such as at anorganization's information technology center or at an employmentrecommendation service's information technology center or a datawarehouse.

[0066] The pre-hire information 112 can be collected for an applicantwhen the applicant applies for a job or other times. For example, datamay be obtained concerning individuals who have yet to apply foremployment, such as from an employee job search web site or firm. Theresponse data can then be used to predict the probable job effectivenessof an applicant and the results of each prediction. Probable jobeffectiveness can be described, for example in terms of desired criteriaand can include behavioral predictions.

[0067] The electronic device can be placed online in a variety of ways.For example, an external telecommunications data link can be used toupload applicant responses to a host computer and download changes inpre-hire content, administration instructions, data handling measures,and other administration functions.

[0068] A modem connection can be used to connect via a telephone networkto a host computer (e.g., central server), or a URL can be used toestablish a web connection (e.g., via the Internet, an intranet, anextranet, and the like). Another network type (e.g., satellite) can beused. In this way, real-time data collection can be implemented.

[0069] The electronic device 124 can allow an applicant to enter text ornumeric data or select from multiple response options, or register avoice or other biophysical response to a machine administered stimulus.The electronic device 124 can be programmable so that the presentedcontent can be modified, and the presented content can be drawn from aremote source. Such content can include text-based questionnaires,multi-media stimuli, and biophysical stimuli.

[0070] The electronic device 124 can, for example, includecomputer-readable media serving as memory for storing pre-hire contentand administration logic as well as the applicant's response data.Alternatively, such content, logic, and responses can be storedremotely.

[0071] The device 124, as other examples, can include a standardcomputer interface (e.g., display, keyboard, and a pointing device),hand-held digital telecommunication devices, digitally enabled telephonedevices, touch-screen kiosk delivery systems, multi-purpose electronictransaction processors such as Automated Teller Machines, travelreservation machines, electronic gaming machines, and biophysicalapparatus such as virtual reality human interface equipment andbiomedicial devices.

[0072] Further, pre-hire information can include geographic elements,allowing geographical specialization (e.g., by region, county, state,country, or the like).

[0073] Post-hire information can similarly be collected in a variety ofways from a variety of sources, including evaluations, terminationinformation, supervisor ratings, payroll information, and directmeasures such as sales or units produced, number of accidents, and thelike.

[0074] For example, after an employee has been on the job for asufficient time, an evaluation can be made. Alternatively, upontermination of the employee, the employee's supervisor can rate theperson's performance in an exit evaluation or the employee can completean employee exit interview. Such collection can be accomplished byreceiving answers to questions on an electronic device, such as thedevice 124 of FIG. 1.

[0075] Other available measures, such as length of service (e.g.,tenure), sales, unit production, attendance, misconduct, number ofaccidents, eligibility for rehire after termination, and whether theemployee was involuntarily terminated may also be collected. Generally,post-hire information is collected for post-hire outcomes for which aprediction is desired. Such outcomes can, for example, includeperformance or job effectiveness measures concurrent with employment.

EXAMPLE 3 Building a Predictive Model

[0076] A variety of techniques can be used to build one or morepredictive models for predicting post-hire outcomes for a job applicant.The model can take one or more inputs (e.g., pre-hire information) andgenerates one or more outputs (e.g., predicted post-hire outcomes). Forexample, a model can be based on artificial intelligence, such as aneural network, a structural equation, an information theoretical model,a fuzzy logic model, or a neuro-fuzzy model.

[0077]FIG. 9 shows an exemplary method 902 for building a predictivemodel. At 912, information relating to inputs (e.g., pre-hireinformation) is collected. At 914, information relating to outputs to bepredicted (e.g., post-hire information) is collected. Based on theinputs and outputs to be predicted, the model is built at 916.

[0078] When building a model, a variety of various proposed models canbe evaluated, and one(s) exhibiting superior performance can be chosen.For example, various types of feed-forward neural networks (e.g., backpropagation, conjugate gradients, quasi-Newton, Levenberg-Marquardt,quick propagation, delta-bar-delta, linear, radial basis function,generalized regression network [e.g., linear], and the like) can bebuilt based on collected pre- and post-hire data and a superior oneidentified and chosen. The proposed models can also be of differentarchitectures (e.g., different number of layers or nodes in a layer). Itis expected that other types of neural network types will be developedin the future, and they also can be used.

[0079] Similar techniques can be used for types of models other thanneural networks. In some cases, trial and error will reveal which typeof model is suitable for use. The advice of an industrial psychologistcan also be helpful to determine any probable interaction effects orother characteristics that can be accounted for when constructingproposed models.

[0080] Various commercially-available off-the-shelf software can be usedfor constructing artificial intelligence-based models of different typesand architectures. For example, NEURALWORKS software (e.g., NEURALWORKSProfessional II/Plus) marketed by NeuralWare of Carnegie, Pa. andSTATISTICA Neural Networks software marketed by StatSoft of Tulsa, Okla.can be used. Any number of other methods for building the model can beused.

[0081] A model can have multiple outputs or a single output. Further,multiple models can be built to produce multiple predictions, such aspredictions of multiple job performance criteria. Also, a model can bebuilt to be geographically specialized by building it based oninformation coming from a particular region, county, state, country, orthe like.

[0082] Occupationally-specialized or education level-specialized modelscan also be constructed by limiting the data used to build the model toemployees of a particular occupation or educational level.

[0083] One possible way of building a neural network is to divide theinput data into three sets: a training set, a test set, and a hold-outset. The training set is used to train the model, and the test set isused to test the model and possibly further adjust it. Finally, thehold-out set is used as a measure of the model's ability to generalizelearned pattern information to new data such as will be encountered withthe model begins processing new applicants. For example, a coefficient(e.g., 0.43) can be calculated to indicate whether the model is validbased on its ability to predict values of the hold-out set. Variousphenomenon related to neural networks, such as over-training can beaddressed by determining at what point during training the neuralnetwork indicates best performance (e.g., via a test set).

[0084] Identifying a superior model out of proposed models can beachieved by ranking the models (e.g., by measuring a validitycoefficient for a hold-out set of data). During the ranking process,particular types (e.g., neural network or fuzzy logic) or architectures(e.g., number of hidden nodes) may emerge as fruitful for furtherexploration via construction of other, similar proposed models.

EXAMPLE 4 Identifying Ineffective Predictors

[0085] Ineffective (e.g., non-predictive or low-predictive) predictorscan be identified. For example, using an information-theory-basedtechnique called “information transfer,” pre-hire content can beidentified as ineffective. Generally, an ineffective predictor is apredictor that does not serve to effectively predict a desired jobperformance criterion. For example, answers to a particular question mayexhibit a random relationship to a criterion and simply serve as noisein data.

[0086] One technique for identifying ineffective predictors is toconsider various sets of permutations of predictive items (e.g., answersto job application questions A, B, C, A & B, A & C, B & C, and A & B &C) and evaluate whether the permutation set is effective. If an item isnot in any set of effective predictors, the item is identified asineffective. It is possible that while an item alone is ineffective, itis effective in combination with one or more other items. Additionalfeatures of information transfer-based techniques are described ingreater detail below.

[0087] After predictors are identified as ineffective, various actionscan be taken, such as omitting them when constructing a model orremoving corresponding questions from a job application. Or, anindication can be provided that information relating to such predictorsno longer need be collected.

EXAMPLE 5 Building a Model Based on Having Identified IneffectivePredictors

[0088] Predictors identified as ineffective can be ignored when buildinga model. In other words, one part of the model-building process can bechoosing inputs for the model based on whether the inputs are effective.

[0089] Reducing the number of inputs can reduce the complexity of themodel and increase the accuracy of the model. Thus, a more efficient andeffective model-building process can be achieved.

EXAMPLE 6 Exemplary Model

[0090]FIG. 10 shows a simple exemplary predictive model 1002 withpredictive inputs IN₁, IN₂, IN₃, IN₄, and IN₅. Various weights a₁, a₂,a₃, a₄, and a₅ can be calculated during model training (e.g., viaback-propagation). The inputs are used in combination with the weightsto generate a predicted value, OUT₁. For example, the inputs might beanswers to questions on a job application, and the predicted value mightbe expected job tenure.

[0091] A predictive model can estimate specific on-the-job behaviorsthat have been described for validation analysis in mathematical terms.Although a two-layer model is shown, other numbers of layers can beused. In addition, various other arrangements involving weights andcombinations of the elements can be used. In fact, any number of otherarrangements are possible.

EXAMPLE 7 Refining a Model

[0092] Predictors identified as ineffective can be removed from pre-hirecontent. For example, if a question on a job application is found to bean ineffective predictor for desired job performance criteria, thequestion can be removed from the job application. Additional questionscan be added (these, too, can be evaluated and possibly removed later).

[0093] New pre-hire information can be collected based on the refinedpre-hire content. Then corresponding new post-hire information can becollected. Based on the new information, a refined model can be built.Such an arrangement is sometimes called “performance-driven systematicrotation of pre-hire content.”

[0094] In this way, questions having little or no value can be removedfrom an employment application, resulting in a shorter but moreeffective application. Predictive content can be identified by placing aquestion into the pool of questions and monitoring whether it isidentified as ineffective when a subsequent model is constructed.

[0095] Model refinement can also be achieved through increased samplesize, improvements to model architecture, changes in the model paradigm,and other techniques.

[0096] A system using the described refinement process can be said toexhibit adaptive learning. One advantage to such an arrangement is thatthe system can adapt to changing conditions such as changing applicantdemographics, a changing economy, a changing job market, changes in jobcontent, or changes to measures of job effectiveness.

EXAMPLE 8 Exemplary Refined Model

[0097]FIG. 11 shows a simple exemplary refined predictive model 1102. Inthe example, it was determined that IN₄ and IN₅ were ineffectivepredictors, so the content (e.g., question) related to IN₄ and IN₅ wasremoved from the corresponding employment application. Based on thefinding that IN₄ and IN₅ were not effective predictors, they were notincluded in the model deployed at that time. A set of new questions wasadded to the employment application.

[0098] When selecting new questions, it may be advantageous to employthe services of an industrial psychologist who can evaluate the job anddetermine appropriate job skills. The psychologist can then determine anappropriate question to be asked to identify a person who will fit thejob.

[0099] Subsequently, after pre-hire and post-hire information for anumber of employees was collected, the new model 1102 was generated fromthe collected information. Two of the new questions were found to beeffective predictors, so they was included in the refined model as IN₈and IN₉. IN₄ and IN₅ do not appear because they had been earlier foundto be ineffective predictors.

EXAMPLE 9 Prediction Types

[0100] A predictive model can generate a variety of prediction types.For example, a single value (e.g., “36 months” as a likely term ofemployment) can be generated. Or, a range of values (e.g., “36-42months” as a likely range of employment term) can be generated. Or, arank (e.g., “7 out of 52” as how this applicant ranks in tenure ascompared to 52 other applicants) can be generated.

[0101] Further, probabilities can be generated instead of or in additionto the above types. For example, a probability that an individual willbe in a certain range can be generated (e.g., “70% -36 or more months”).Or, a probability of a certain value can be generated (“5% -0accidents”). Or, probability of membership in a group can be generated(e.g., “75% involuntarily terminated”).

[0102] Various combinations and permutations of the above are alsopossible. Values can be whatever is appropriate for the particulararrangement.

EXAMPLE 10 Predicted Outcomes

[0103] Predicted post-hire outcomes can be any of a number of metrics.For example, number of accidents, sales level, eligibility for rehire,voluntary termination, and tenure can be predicted. There can be variousmodels (e.g., one for each of the measurements) or one model can predictmore than one. The predicted outcomes can be job performance criteriaused when making a hiring recommendation.

EXAMPLE 11 Hiring Recommendation

[0104] After determining the suitability of the individual foremployment by the employer, based on one or more predictions generatedby one or more models, a hiring recommendation can be made. Therecommendation can be provided by software.

[0105] The recommendation can include an estimate of future behavior andresults can be reported in behavioral terms. Alternatively, an employermight indicate the relative importance of predicted outcome values, suchas a specific set of job performance criteria. Such information can becombined with generated predicted outcomes to generate an overall score.Applicants having a score over a particular threshold, for example, canbe identified as favorable candidates. Further evaluation (e.g., askills test or interview) may or may not be appropriate.

EXAMPLE 12 Payroll-Based Information Collection

[0106] A problem can arise when collecting post-hire information. Forexample, it may be difficult to achieve high compliance rates for exitinterviews. Also, collection of information relating to terminationdates and reasons for termination may be sporadic.

[0107] Post-hire information can be generated by examining payrollinformation. For example, a system can track whether an employee hasbeen dropped from the payroll. Such an event typically indicates thatthe employee has been terminated. Thus, the employee's tenure can bedetermined by comparing the termination date with the employee's hiredate. Further, available payroll information might indicate whether anemployee was voluntarily or involuntarily terminated and whether or notthe employee is eligible for rehire and why the termination occurred.Still further, the payroll information can indicate a job change (e.g.,a promotion).

[0108] Thus, much post-hire information can be commonly collected basedon payroll information, and a higher sample size can be achieved. Anexemplary arrangement 1202 for collecting such information is shown inFIG. 12. In the example, the payroll information 1212 is accessible by apayroll server 1222. Communication with the payroll server 1222 can beachieved over a network 1242 (e.g., via the Internet or anothernetwork). The server 1242 receives information from the payroll server1222 via the network 1232 (e.g., via any number of protocols, such asFTP, email, and the like). The information is then stored in thepost-hire information database 1252. For example, payroll informationcan be scheduled for automatic periodic sending or may be sent uponinitiation by an operator.

[0109] Although an online arrangement is shown, the information can alsobe provided manually (e.g., via removable computer-readable media). Insome cases, the information may need to be reformatted so it matches theformat of other data in the database 1252.

EXAMPLE 13 Exemplary Implementations

[0110] In various implementations of the technologies, acomputer-implemented system can be provided that collects pre-hireapplicant information used to assess suitability for employment inspecific jobs. The computer system can also collect post-hire measuresof the job effectiveness of employees hired using the system.

[0111] The pre-hire and post-hire information can then be converted andstored electronically as numeric data where such data can be logicallyquantified. Artificial intelligence technology and statistical analysiscan be used to identify patterns within the pre-hire data that areassociated with patterns of job effectiveness stored in the post-hiredata. Pre-hire data patterns with significant associations withdifferent post-hire patterns are then converted to mathematical models(e.g., data handling routines and equations) representing the observedrelationships.

[0112] Following the development of interpretive algorithms thatoperationalize the pattern relationships observed in a sample ofcomplete employment cycles, the pre-hire data collection system can thenbe re-programmed to run such interpretive formulas on an incoming datastream of new employment applications. Formula results can beinterpreted as an estimate of the probable job effectiveness of newapplicants for employment based on response pattern similarity to others(e.g., employees). Interpretive equation results can be reported inbehavioral terms to hiring managers who can use the information toidentify and hire those applicants whose estimated job performance fallswithin an acceptable range.

[0113] The system can be capable of adaptive learning, or the ability tomodify predictive models in response to changing data patterns. Adaptivelearning can be operationalized using artificial intelligencetechnologies, short cycle validation procedures and performance-drivenitem rotation. The validation cycle can be repeated periodically as newemployment histories are added to the database. With successivevalidation cycles, pre-hire predictor variables that have little or norelationship to job effectiveness can be dropped. New item content canreplace the dropped items. Predictive variables can be retained and usedby interpretive algorithms until sufficient data has accumulated tointegrate the new predictors into the next generation interpretivealgorithm. The outdated algorithm and associated records can be archivedand the new model deployed. Adaptive learning can enable evolutionaryperformance improvement, geographic specialization, and shorter, moreaccurate pre-hire questionnaires.

EXAMPLE 14 Criterion Validation

[0114] Criterion validation includes discovering and using measures ofindividual differences to identify who, out of a group of candidates, ismore likely to succeed in a given occupation or job. Individualdifferences are measures of human characteristics that differ acrossindividuals using systematic measurement procedures. Such measuresinclude biographic or life history differences, standardized tests ofmental ability, personality traits, work attitudes, occupationalinterests, work-related values and beliefs, and tests of physicalcapabilities, as well as traditional employment-related information,such as employment applications, background investigation results,reference checks, education, experience, certification requirements, andthe like.

[0115] Criterion validation includes the research process used todiscover how these measures of individual differences relate to acriterion or standard for evaluating the effectiveness of an individualor group performing a job. Typical measures of job effectiveness includeperformance ratings by managers or customers, productivity measures suchas units produced or dollar sales per hour, length of service,promotions and salary increases, probationary survival, completion oftraining programs, accident rates, number of disciplinary incidents orabsences, and other quantitative measures of job effectiveness. Any ofthese measures ofjob effectiveness and others (e.g., whether anapplicant will be involuntarily terminated, and the like) can bepredicted via a model.

[0116] Pre-hire metrics, including those listed above, calledpredictors, can be analyzed in relation to each criterion to discoversystematic co-variation. A common statistic used to summarize suchrelationships is the Pearson Product Moment Correlation coefficient, orsimply the validity coefficient. If a predictor measure is found tocorrelate with a criterion measure across many individuals in avalidation sample, the predictor is said to be “valid,” that ispredictive of the criterion measure. Valid predictors (e.g., pre-hireinformation) that correlate with specific criteria, such as post-hiremeasures (e.g., including concurrent performance measures) are then usedin the evaluation of new candidates as they apply for the same orsimilar jobs. Individual differences in temperament, ability, and othermeasures can have profound and measurable effects on organizationaloutcomes.

[0117] In employee selection, an independent (e.g., “predictor”)variable can be any quantifiable human characteristic with a measurablerelationship to job performance. Physical measurements, intelligencetests, personality inventories, work history data, educationalattainment, and other job-related measures are typical. The dependent(e.g., “criterion”) variable can be defined as a dependent or predictedmeasure for judging the effectiveness of persons, organizations,treatments, or predictors of behavior, results, and organizationaleffectiveness.

[0118] In general, measures of job performance include objective numericdata, such as absenteeism, accident rates, unit or sales productivitycan be readily verified from direct observation and are sometimes called“hard” measures. Objective measures of job performance may be availablefor only a small set of narrowly-defined production and otherbehaviorally-specific jobs. In the absence of hard measurement, opiniondata such as performance ratings by managers can be used for the samepurpose.

[0119] Establishing the criterion validity of a selection test or groupof tests can include informed theory building and hypothesis testingthat seeks to confirm or reject the presence of a functionalrelationship.

EXAMPLE 15 Artificial Intelligence Techniques

[0120] Artificial intelligence can attempt to simulate humanintelligence with computer circuits and software. There are at leastthree approaches to machine intelligence: expert systems, neuralnetworks, and fuzzy logic systems. Expert systems can capture knowledgeof human experts using rule-based programs to gather information andmake sequential decisions based on facts and logical branching. Thesesystems involve human experts for constructing the decision modelsnecessary to simulate human information processing. Expert systems canbe used to standardize complex procedures and solve problems withclearly defined decision rules.

[0121] Neural networks (also commonly called “neural systems,”“associative memories,” “connectionist models,” “parallel distributedprocessors,” and the like) can be computer simulations ofneuro-physiological structures (e.g., nerve cells) found in nature.Unlike expert systems, artificial neural networks can learn byassociation or experience, rather than being programmed. Like theirbiological counterparts, neural neworks form internal representations ofthe external world as a result of exposure to stimuli. Once trained,they can generalize or make inferences and predictions about data thatthey have not been exposed to before. Neural networks are able to createinternal models of complex, nonlinear multivariate relationships, evenwhen the source data is noisy or incomplete. It is this capacity tofunction with uncertain or fuzzy data that makes a neural processorvaluable in the real world.

[0122] Fuzzy computation includes a set of procedures for representingset membership, attributes, and relationships that cannot be describedusing single point numeric estimates. Fuzzy systems can allow computersto represent words and concepts such as vagueness, uncertainty, anddegrees of an attribute. Fuzzy systems can allow computers to representcomplex relationships and interactions between such concepts. They canalso be a useful tool for describing human attributes in terms that acomputer can process. Fuzzy concepts and fuzzy relationship models canbe used in an employee selection system to represent predictor-criterioninteractions when such relationships are supported by analysis of theavailable data.

[0123] Neuro-fuzzy technology is a hybrid artificial intelligencetechnique employing the capabilities of both neural network learning andfuzzy logic model specification. In an employee selection system,predictor-criterion relationships can be described initially as a fuzzymodel and then optimized using neural network training procedures. Inthe absence of evident explanatory predictor-criterion relationships,unspecified neural networks can be used until such relationships can beverified.

[0124] Genetic algorithms can represent intelligent systems bysimulating evolutionary adaptation using mathematical procedures forreproduction, genetic crossover, and mutation. In an employee selectionsystem, genetic algorithm-based data handling routines can be used tocompare the prediction potential of various combinations of predictorvariables to optimize variable selection for model development.

[0125] Information theoretic based feature selection can be based oninformation theory. Such a technique can use measures of informationtransmission to identify relations between independent and dependentvariables. Since information theory does not depend on a particularmodel, relation identification is not limited by the nature of therelation. Once the identification process is complete, the set ofindependent variables can be reduced so as to include only thosevariables with the strongest relationship to the dependent variables.

[0126] Such a pre-filtering process facilitates the modeling process byremoving inputs which are (e.g., for the most part) superfluous andwould therefore constitute input noise to the model. A reduction in thedimensionality of the input vector to the model also reduces thecomplexity of the model and in some cases (e.g., neural networks),greatly reduces the computational expense involved in model generation.

[0127] Information theoretic-based modeling techniques such asreconstructability analysis can be used in an employee selection system.Such techniques use informational dependencies between variables toidentify the essential relations within a system. The system is thenmodeled by reproducing the joint probability distributions for therelevant variables. The benefits of such modeling techniques includethat they do not depend on a model and can emulate both deterministicand stochastic systems.

[0128] An employee selection system can include adaptive learningtechnology. Such a system can be constructed as a hybrid artificialintelligence application, based in part on various (or all) of the aboveartificial intelligence technologies. Expert systems can be employed tocollect and process incoming and outgoing data, transfer data betweensub-systems internally and in model deployment. Neural networks can beused for variable selection, model development, and adaptive learning.Fuzzy set theory, fuzzy variable definition, and neuro-fuzzy procedurescan be used in variable specification, model definition, and refinement.Genetic algorithm techniques can be used in variable selection, neuralnetwork architecture configuration and model development and testing.Information theoretic feature selection and modeling techniques can beused in data reduction, variable selection, and model development.

EXAMPLE 16 Electronic Repository System

[0129] Externally-collected data can be sent to an in-boundcommunications sub-system that serves as a central repository ofinformation. Data can be uploaded via a variety of techniques (e.g.,telephone lines, Internet, or other data transfer mechanisms). Thein-bound communications sub-system can include a set of softwareprograms to perform various functions.

[0130] For example, the sub-system can receive incoming data fromexternal data collection devices. The incoming data can be logged with adate, time and source record. Data streams can be stored to a backupstorage file.

[0131] After data reception, the subsystem can respond to the sourcedevice with a text message indicating that transmission was successfulor unsuccessful; other messages or instructions can be provided. Thedata stream can be transferred to a transaction monitor (e.g., such asthat described below) for further processing.

[0132] The subsystem can also download machine-specific executable codeand scripting files to external data collection devices when changes tothe user-interface are desired. The download transmissions can be loggedby date, time, and status and the external device's response recorded.

EXAMPLE 17 Transaction Monitor

[0133] A transaction monitor can serve as an application processingsystem that directs information flow and task execution between andamong subsystems. The transaction monitor can classify incoming andoutgoing data streams and launch task-specific sub-routines usingmulti-threaded execution and pass sub-routine output for furtherprocessing until transactions (e.g., related to data streams) have beensuccessfully processed.

[0134] A transaction monitor can perform various functions. For example,the transaction monitor can classify data streams or sessions astransactions after transmission to an in-bound communicationssub-system. Classification can indicate the processing tasks associatedwith processing the transaction.

[0135] Data can be parsed (e.g., formatted into a pre-defined structure)for additional processing and mapped to a normalized relational database(e.g., the applicant database described below). Data elements can bestored with unique identifiers into a table containing similar data fromother sessions.

[0136] Session processing task files can be launched to process parseddata streams. For example, an executable program (e.g., C++ program,dynamic link library, executable script, or the like) can performvarious data transmission, transformation, concatenation, manipulationor encoding tasks to process the sessions.

[0137] Output from session processing tasks can then be formatted forfurther processing and transmission to external reporting devices (e.g.,at an employer's site). For example, the imaging and delivery sub-systemdescribed below can be used.

EXAMPLE 18 Applicant Database

[0138] A relational database can store pre- and post-employment data forsession transactions that are in process or were received and recentlyprocessed. As individual session records age, they can be systematicallytransferred to another storage database (e.g., the reports databasedescribed below).

[0139] Both databases can consist of electronically-stored tables madeup of rows and columns of numeric and text data. In general, rowscontain identifier keys (e.g., unique keys) that link elements of aunique session to other data elements of that session. Columns can holdthe component data elements. Unique session data can be stored acrossmany tables, any of which may be accessed using that session's uniqueidentification key.

[0140] An arrangement of three basic types of data can be used for theapplicant database. First, standard pre-hire application information(e.g., name, address, phone number, job applied for, previousexperience, references, educational background, and the like) can bestored. Also, included can be applicant responses to psychological orother job-related assessments administered via an external datacollection device (e.g., the electronic device 124 of FIG. 1).

[0141] Second, post-hire data about the job performance of employeesafter being hired can be stored. Such data can include, for example,supervisor opinion ratings about the employee's overall job performanceor specific aspects of the employee's job effectiveness. Quantitativeindicators about attendance, sales or unit production, disciplinaryrecords and other performance measures may also be collected.

[0142] Third, employer-specific information used to process transactionscan be stored. Such data can include information for sending anappropriate electronic report to a correct employer location,information related to downloading user interface modifications tospecific data collection devices, and information for general managementof information exchange between various sub-systems. For example,employer fax numbers, URL's, email accounts, geographic locations,organizational units, data collection unit identifier, and the like canbe stored.

[0143] Other information or less information can be stored in thedatabase. Further, the database may be broken into multiple databases ifdesired.

EXAMPLE 19 Reports Database

[0144] A reports database can be a relational database serving as acentral repository for records processed by the applicant database.Applicant records for applicants not hired can be deleted. Applicantrecords for applicants aged over a certain client-specified recordretention time limit can be deleted.

[0145] The reports database can be used as a source for the data used ingenerating, printing, or posting corporate reports (e.g., such as thosedescribed below). Such data can include client-specific records ofemployment applications received for recent reporting periods, pluspre-hire predictor and post-hire criterion performance data.

EXAMPLE 20 Corporate Reports

[0146] Useful information can be collected in the course of operating ahiring recommendation system. For example, information about applicantflow, hiring activity, employee turnover, recruiting costs, number ofvoluntary terminations, applicant and employee characteristics and otheremployee selection metrics can be collected, stored, and reported.

[0147] Standardized reports can be provided to employers via printedreports, fax machines, email, and secure Internet web site access.Source data can come from the reports database described above. Customreports can also be generated.

EXAMPLE 21 Sample Size Monitor

[0148] A sample size monitor can be provided as a computer program thatmonitors the quality and quantity of incoming data and provides anindication when a sufficient number or predictor-criterion paired caseshave accumulated. For example, employer-specific validation data can betransferred to a model development environment upon accumulation ofsufficient data.

[0149] The program can use an expert system decision rule base to keeptrack of how many complete employee life cycle histories are in areports database. In addition, the software can examine and partitionindividual records that may be unusable due to missing fields, corrupteddata, or other data fidelity problems. Using pre-defined sample sizeboundaries, the software can merge available pre- and post-hire datatransfer and transfer a file to the validation queue (e.g., the queuedescribed below).

EXAMPLE 22 External Service Providers

[0150] A system can interface with other online data services ofinterest to employers. Using a telecommunication link to third partyservice computers, a transaction monitor can relay applicant informationto trigger delivery of specialized additional pre-hire data which canthen be added to an applicant database and used in subsequent analysisand reporting. Such services can include, for example, online workopportunity tax credit (WOTC) eligibility reporting, online socialsecurity number verification, online background investigation results asindicated by specific jobs, and psychological assessment results,including off-line assessment. Such services are represented in FIG. 1as the third party service 126.

EXAMPLE 23 Validation Queuing Utility

[0151] Validation queuing utility software can be provided to serve as atemporary storage location for criterion validation datasets that havenot yet been processed in a model development environment (e.g., such asthat described below). Datasets can be cataloged, prioritized, andscheduled for further processing using predefined decision rules. Whenhigher priority or previously-queued datasets have been processed, thefile can be exported to the analysis software used for modeldevelopment.

EXAMPLE 24 Model Development Technique

[0152] Model development can result in the creation of a model thatrepresents observed functional relationships between pre-hire data andpost-hire data. Artificial intelligence technologies can be used todefine and model such relationships. Such technologies can includeexpert systems, neural networks and similar pattern function simulators,fuzzy logic models, and neuro-fuzzy predictive models.

[0153] Various procedures can be implemented. For example, thedistribution of pre-hire variables (sometimes called “independent” or“predictor variables”) can be analyzed in relation to the distributionof post-hire outcome data (sometimes called “dependent” or “criterionvariables”).

[0154] Using statistical and information theory derived techniques, asubset of predictor variables can be identified that show informationtransfer (e.g., potential predictive validity) to one or more criterionvariables.

[0155] An examination of joint distributions may result in theformalization of a fuzzy theoretical model and certain predictors may betransformed to a fuzzy variable format.

[0156] If an obvious theoretical model does not emerge from thisprocess, the remaining subset of promising variables can be categorizedand transformed for neural network training. Non-useful (e.g.,ineffective) predictor variables can be dropped from further analysis.

[0157] The total sample of paired predictor-criterion cases (e.g.,individual employee case histories) can be segmented into threenon-overlapping sub-samples with group membership being randomlydefined. Alternate procedures, such as randomized membership rotationmay also be used to segment the data.

[0158] A training set can be used to train a neural network orneuro-fuzzy model to predict, classify, or rank the probable criterionvalue associated with each instance of predictor input variables. A testset can be used to evaluate and tune the performance (e.g., predictiveaccuracy) of models developed using the training set. A hold-out orindependent set can be used to rank trained networks by their ability togeneralize learning to unfamiliar data. Networks with poor predictiveaccuracy or low generalization are dropped from further development.

[0159] Surviving trained models can then be subjected to additionaltesting to evaluate acceptability for operational use in employeeselection. Such testing can include adverse impact analysis andselection rate acceptability.

[0160] Adverse impact analysis can evaluate model output fordifferential selection rates or bias against protected groups. Usingindependent sample output, selection rates can be compared acrossgender, ethnicity, age, and other class differences for bias for oragainst the groups. Models which demonstrate differential prediction orimproper bias can be dropped from further development.

[0161] Selection rate acceptability can include evaluation of selectionrates for hire/reject classification models. Selection rates on theindependent sample can be evaluated for stringency (e.g., rejects toomany applicants) or leniency (e.g., accepts too many applicants) andmodels showing these types of errors can be dropped.

[0162] Final candidate networks can be ranked according to theirperformance on test parameters, and the single best model can beconverted to a software program for deployment in a live employeeselection system. The coded program can then be passed to the deploymentand archiving modules (e.g., such as those described below).

[0163] Such an iterative process can be repeated as differentpredictor-criterion relationships emerge. As sufficient data accumulateson specific criterion outcomes, additional predictive models can bedeveloped. Older models can eventually be replaced by superiorperforming models as item content is rotated to capture additionalpredictive variation (e.g., via the item rotation module describedbelow). Sample size can continue to increase. Thus, a system can evolvetoward higher predictive accuracy.

EXAMPLE 25 Model Deployment Technique

[0164] Deployment of a model can include a hiring report modificationand model insertion. The hiring report modification can includemodifications to an imaging and delivery subsystem and an applicantprocessing system (e.g., the above-described transaction monitor).

[0165] To facilitate employer use of model predictions, numeric outputcan be translated into text, number, or graphics that are descriptive ofthe behavior being predicted. Output can be presented to an employer inbehavioral terms.

[0166] When a criterion to be predicted is a number, the exact numericestimate can be couched in a statement or picture clearly describing thepredicted behavior. For example, if the model has produced an estimateof an applicant's probable length of service in days, the hiring reportcan be modified to include a statement such as the following example:

[0167] Based on similarity to former employees, this applicant'sestimated length of service is X days, plus or minus Y days margin oferror.

[0168] X can be the specific number of days that the trained predictivemodel has provided as an estimate of the applicant's probable length ofservices, and Y can be the statistical margin of error in which themajority of cases will tend to fall.

[0169] When the criterion to be predicted is group membership (e.g.,whether or not the applicant is likely to belong to a specific group),the model estimate may be expressed as a probability, or likelihood,that the applicant will eventually be classified in that group. Forexample, if the predictive model has been trained to classify employeeresponse patterns according to the probability that they would beeligible for rehire instead of not being eligible for rehire upontermination, a statement or graphic similar to the following example canbe presented on a hiring report:

[0170] Based on similarity to former and/or current employees, thisapplicant 's probability of being eligible for rehire upon terminationis X percent.

[0171] X can be a probability function expressed as a percentagerepresenting the number of chances in one hundred that the particularapplicant will be eligible for rehire when he or she leaves the company.

[0172] When the criterion produced is a ranking or relative position ina ranked criterion, text or graphic images can be used to convey theapplicant's position in the criterion field. For example, if the modelhas produced an estimate of the probable rank of a sales employee'sannual sales volume compared to past sales employees, a statementsimilar to the following example might be used:

[0173] Based on similarity to former sales employees, this applicant islikely to produce annual sales in the top Xth (e.g, third, quarter,fifth, or the like) of all sales employees.

[0174] X can refer to the ranking method used to classify the criterionmeasure.

[0175] Such text-based reporting methods as described above can besummarized, illustrated with, appended to, or replaced by graphic imagesrepresenting the behavioral information. For example, charts, graphs,images, animated images, and other content format can be used.

[0176] Applicant processing system model insertion can be accomplishedby embedding a coded model in the application processing conducted by atransaction monitor after the format of the predictive output has beendetermined. Data handling routines can separate model input variablesfrom the incoming data stream. The inputs can be passed to thepredictive model and be processed. The output of the model can then beinserted or transformed into a reporting format as described above andadded to a hiring report transmission.

EXAMPLE 26 Validation Archives

[0177] As a new model is deployed, the replaced model can be transferredto an archive storage. The archive can also record applicants processedby the old model. Such an archive can be useful if reconstruction ofresults for a decommissioned model is desired for administrative orother reasons.

EXAMPLE 27 Exemplary Item Rotation Technique

[0178] An item rotation module can be implemented as a software programand database of predictor item content. The item rotation module can beused to systematically change pre-hire content so that useful predictorvariables are retained while non-useful (e.g., ineffective) predictorscan be replaced with potentially useful new predictors.

[0179] Adaptive learning includes the ability of a system to improveaccuracy of its behavioral predictions with successive validationcycles. Iterative neural network and neuro-fuzzy model development andperformance-driven item rotation can be used to facilitate adaptivelearning.

[0180] As part of a validation analysis for a model, predictor variables(e.g., pre-hire questions or items) predictive of a criterion measurecan be identified. At the same time, other predictors with little or nomodeling utility (e.g., ineffective predictors) can be identified.

[0181] Performance-driven item rotation includes the practice ofsystematically retaining and deleting pre-hire content so that itemcontent with predictive utility continues to serve as input forbehavioral prediction with the current predictive model and items withlittle or no predictive utility are dropped from the content. New,experimental item content can be inserted into the content and responsepatterns can be recorded for analysis in the next validation cycle. Suchrotation is shown in Tables 1 and 2. TABLE 1 Item Content DuringValidation Cycle #1 Item Status You help people a lot Ineffective Youtease people until they get mad Ineffective You have confidence inyourself Effective You would rather not get involved in Ineffectiveother's problems Common sense is one of your greatest Ineffectivestrengths You prefer to do things alone Effective You have no fear ofmeeting people Effective You are always cheerful Ineffective 24 × 7 = ?Ineffective You get mad at yourself when you make Ineffective mistakesHow many months were you at your last Effective job?

[0182] TABLE 2 Item Content After Validation Cycle #1 Item Status Manypeople cannot be trusted New experimental item You are not afraid totell someone off New experimental item You have confidence in yourselfEffective-retained You try to sense what others are thinking Newexperimental item and feeling You attract attention to yourself Newexperimental item You prefer to do things alone Effective-retained Youhave no fear of meeting people Effective-retained You can wait patientlyfor a long time New experimental item You say whatever is on your mindNew experimental item Background check item New experimental item Howmany months were you at your last Effective-retained job?

[0183] The content shown in Table 1 has been refined to be that shown inTable 2, based on the effectiveness of the predictor items. Newexperimental items have been added, the effectiveness of which can beevaluated during subsequent cycles.

[0184] As successive validation cycles are completed and non-predictiveitem content is systematically replaced with predictive item content,overall validity improves. After multiple validation cycles, the resultcan be a shorter pre-hire questionnaire comprised ofcurrently-performing predictive input and a few experimental items beingvalidated in an on-going process for system evolution toward higherpredictive accuracy.

EXAMPLE 28 Imaging and Delivery Subsystems

[0185] Imaging and delivery subsystems can assemble input from applicantprocessing to create an electronic image that resembles a traditionalemployment application that can be transmitted to an employer's hiringsite via external data devices (e.g., fax machine, computer with emailor web access, hand-held devices, digitally enabled telephones,printers, or other text/graphics imaging devices). Hiring reports canalso be delivered as hard copy via mail or other delivery services.

EXAMPLE 29 Hire Site Report Reception

[0186] Hiring managers can receive an electronic report that can beprinted or simply saved in electronic format. The entire applicationprocess can occur in real-time or batch mode (e.g., overnight bulkprocessing). Real-time processing can result in hiring report receptionminutes after pre-hire data is uploaded. Such rapid report reception canbe an advantage of the system.

EXAMPLE 30 Exemplary Combination of Elements

[0187] The various above-described elements can be combined in variouscombinations and sub-combinations to construct a system. For example,FIG. 13 shows an exemplary combination of elements.

[0188] Pre-hire and post-hire data collection elements 1312 can send,via the incoming communications subsystem 1316, information to thetransaction monitor 1318. The information can be stored in the applicantdatabase 1322 while processed and then stored in the reports database1324. The reports database 1324 can be used to produce corporate reports1328.

[0189] A sample size monitor 1332 can monitor the reports database 1324and send information, via the validation queue 1338, to the predictivemodel development environment 1342. Models from the developmentenvironment 1342 can be sent for model deployment 1348, including hiringreport modification and model insertion.

[0190] Archived models can be sent to the validation archives 1352, andan item rotation module 1358 can track rotation of predictive content.Imaging and delivery subsystems 1372 can deliver hire site reports 1378.

[0191] External service providers 1388 can interface with the system1302 to provide a variety of data such as applicant pre-hire information(e.g., background verification, credit check information, socialsecurity number verification, traffic and criminal information, and thelike).

[0192] Fewer or additional elements can be included in a system.

EXAMPLE 31 Exemplary Process Overview

[0193] The various techniques described above can be used in a processover time. In such a process, adaptive learning can improve employeeselection with successive validation cycles as sample size increases andpredictor input systematically evolves to capture more criterionrelationships and higher predictor-criterion fidelity. An example isshown in FIGS. 14A-14D.

[0194]FIG. 14A shows a first cycle 1402. For example, when an employerfirst begins to use a system, applicants enter pre-hire application andassessment responses using external data collection devices. The datacan be stored and processed as described above, except that as of yet nobehavioral predictions appear on the hiring report because a sufficientnumber of employee histories has not yet been captured by the system.

[0195] As employee job performance measures are taken, employees leaveand complete exit interviews and their managers complete an exitevaluation, or payroll information is collected also using the externaldata collection devices, employee histories are added to the database.The rate of data accumulation is a function of how quickly people apply,are hired, and then terminate employment. An alternative to capturingpost-hire job performance data upon termination is to collect similardata on the same population prior to termination on a concurrent basis.In the example, the size of the validation database is small, there isno adaptive learning, there are no predictive models, and there are nobehavioral predictions.

[0196] When a sufficient sample of employee histories is available,validation and predictive modeling can occur. Following modeldevelopment, the second validation cycle 1422 can begin as shown in FIG.14B. Ineffective pre-hire variables are dropped or replaced with newcontent and the pre-hire application is modified. Applicant andterminating employee processing continues and more employee historiesare added to the database. In the example, the validation database ismedium, there is at least one predictive model, and there is at leastone behavioral prediction (e.g., length of service or tenure).

[0197] A third validation cycle 1442 is shown in FIG. 14C. Initially,predictive modeling might be limited to behavioral criteria commonlyobserved, such as length of service, rehire eligibility, or jobperformance ratings because sample sufficiency occurs first with suchcommon measures. Other less frequently occurring data points (e.g.,misconduct terminations) typically accumulate more slowly. As managersbegin using the behavioral predictions to select new employees, thecomposition of the workforce can begin to change (e.g., newer employeesdemonstrate longer tenure, higher performance, and the like).

[0198] As usable samples are obtained for different criteria (e.g.,post-hire outcomes), new models are developed to predict thesebehaviors. Older predictive models can be replaced or re-trained toincorporate both new item content from the item rotation procedure andadditional criterion variation resulting from the expanding number ofemployee histories contained in the validation database. In the example,the validation database is large, there are differentiated models, and anumber of behavioral predictions (e.g., tenure, early quit, andeligibility for rehire).

[0199] Fourth and subsequent validation cycles 1462 are shown in FIG.14D. Multiple iterations of the validation cycle using larger and largervalidation samples result in multiple complex models trained to producesucessively-improving behavioral prediction across the spectrum ofmeasurable job-related outcomes (e.g., eligibility for rehire, tenure,probable job performance, probability of early quit, job fit,misconduct, and the like). In the example, the validation database isvery large, there are complex, differentiated models, and manybehavioral predictions.

[0200] The behavioral predictions can become more accurate the longerthe system is in place. If used consistently over time, the workforcemay eventually be comprised entirely of employees selected on the basisof their similarity to successful former employees. Continued use of theadaptive learning employee selection technology can be expected toproduce positive changes in the global metrics used to assess workforceeffectiveness. Such metrics include lower rates of employee delinquency(e.g., theft, negligence, absenteeism, job abandonment, and the like),higher rates of productivity (e.g., sales, unit production, servicedelivery, and the like), longer average tenure and reduced employeeturnover, and higher workforce job satisfaction and more effectiveemployee placement.

EXAMPLE 32 Exemplary Process Overview

[0201]FIG. 15 is a process flow diagram illustrating an exemplaryprocess 1502 for an employment suitability prediction system. At 1512,data is collected. Such collection can be accomplished in a wide varietyof ways. For example, electronic data collection units can bedistributed, or a URL can be used by employment applicants.

[0202] Electronic versions of a standard employment application or testscan be deployed. Also, post-hire data collection can be accomplished bydeploying post-hire data collection questionnaires and via payroll datatransfer. Also, manager feedback report apparatus (e.g., fax backreports or e-mail report of results) can be deployed so managers canreceive information such as hiring recommendations. The service can thenbe implemented, and data collection can begin.

[0203] At 1522, feature selection can take place. Pre-hire applicationrecords can be extracted from an applicant processing system, andpost-hire outcome data can be extracted from a reports database. Pre-and post-data can be sorted and matched from both sources to create amatched predictor-criterion set. Information theoretic feature selectioncan be run to identify top-ranking predictive items based on informationtransmission (e.g., mutual information). Item data characterized bymarginal mutual information can be deleted and a distilled predictivemodeling dataset can be saved.

[0204] At 1532, model development can take place. The distilledpredictive modeling dataset can be randomized and partitioned intotraining, testing, and verification subsets. A group of models (e.g.,neural networks) that meet performance criteria thresholds can be builtby experimenting with multiple neural network paradigms, architectures,and model parameters.

[0205] The models can be tested for their ability to generalize (e.g.,apply learned pattern information from training and test sets to theverification dataset). Non-generalizing models can be discarded and thesurviving models can be saved.

[0206] Surviving models can be tested for differential prediction,adverse impact and other anomalies. Biased nets can be discarded.Unbiased models can be ranked and saved.

[0207] At 1542, model deployment can take place. The top-performingsurviving model can be converted to software command code. The code canbe integrated into a custom session processing task which executes modelprocessing and exports the output to an imaging program and hiringreport generator.

[0208] The new session processing task can be tested for appropriatehandling and processing of the incoming data stream values in a softwaretest environment. The session processing task code can be refined anddebugged if necessary. Then, the new task can be deployed in anoperational applicant processing system.

[0209] At 1552, performance tuning can take place. Data collection cancontinue. Sample size can be monitored as incoming data accumulates.When an update threshold is reached, new cases can be added to thematched predictor-criterion set by repeating feature selection 1522.Item content can be revised using a performance driven item rotationprocedure (e.g., replace or remove survey items with marginalinformation transmission). Model development 1532, model deployment1542, and performance tuning 1552 can then be repeated.

EXAMPLE 33 Effectiveness of a Model

[0210] Real-time electronic collection of data and sample size-drivenrefinement of models can result in high model effectiveness. Forexample, FIG. 16 shows a graph 16 in which effectiveness 1622 of areference system is shown. As conditions change over time, theeffectiveness 1622 of the system decreases. The mean effectiveness 1624is also shown.

[0211] As system employing real-time electronic data collection andsample size-driven model refinement can exhibit the effectiveness 1632as shown. As the model is refined, the effectiveness of the modelincreases over time. Thus, the mean effectiveness 1634 is greater,resulting in a more effective system.

EXAMPLE 34 Exemplary Automated Hiring Recommendation Service

[0212] Using various of the technologies, a method for providing anautomated hiring recommendation service for an employer can be provided.Electronic devices can be stationed at employer sites (e.g., retailoutlets). The electronic devices can directly accept pre-hireinformation from job applicants (e.g., answers to questions from a jobapplication). The pre-hire information can then be sent to a remote site(e.g., via a network of telephone connection) for analysis. Anartificial intelligence-based predictive model or other model can beapplied to the pre-hire information to generate an automated hiringrecommendation, which can be automatically sent to the employer (e.g.,via email).

EXAMPLE 35 Exemplary Implementation

[0213] A behavioral prediction model can be developed to generate anestimate of the tenure (length of service in days) to be expected ofapplicants for employment as customer service representatives of anational chain of video rental stores. Such predictions can be based onthe characteristics and behaviors of past employees in the same job atthe same company. Application of the model can result in higher averagetenure and lower employee turnover.

[0214] As a specific example, pre-hire application data used to developthis exemplary model was collected over a period of a year and a halfusing an electronic employment application as administered using screenphones deployed in over 1800 stores across the United States.Termination records of employees hired via the system were received bydownload. Over 36,000 employment applications were received in thereporting period, of which approximately 6,000 resulted in employment.Complete hire to termination records were available for 2084 of theseemployees, and these records were used to develop the model.

[0215] When building the model, definition of system inputs and outputswas accomplished. Independent or predictor variables can be measures ofindividual characteristics thought to be related to a behavior oroutcome resulting from a behavior. In industrial psychology and employeeselection, typical predictor variables might be measures of education,experience or performance on a job-related test. Criterion variables canbe measures of the behavior or outcome to be predicted and might includesales effectiveness, job abandonment, job performance as measured bysupervisor ratings, employee delinquency and other behavioral metrics orcategories.

[0216] In this example, predictor variables are inputs and criterionvariables are outputs. In this research, input variables consist of asubset of the employment application data entered by applicants whenapplying for jobs (see Tables 4 and 5 for a listing of the variablesused in this model). The output or criterion is the number of days thatan employee stayed on the payroll.

[0217] The process of identifying the subset of predictor variables tobe used in a model is sometimes called “feature selection.” While anyinformation gathered during the employment application process may havepredictive value, the set of predictors is desirably reduced as much aspossible. The complexity (as measured by the number of networkconnections) of a network can increase geometrically with the number ofinputs. As complexity increases so can training time along with thenetwork's susceptibility to over-training. Therefore inputs with lesspredictive power can be eliminated in favor of a less complex neuralnetwork model.

[0218] For the tenure prediction model in this illustrative example,information theoretic methods were employed to determine the subset ofinput variables that maximized information transmission between thepredictor set and the criterion. Such an approach can rely on thestatistical theory of independent events, where events p₁, P₂, . . . ,p_(n) are considered statistically independent if and only if theprobability P, that they occur on a given trial is $\begin{matrix}{P = {\prod\limits_{l = 1}^{n}\quad p_{1}}} & (1)\end{matrix}$

[0219] Conversely, the measurement of how much a joint distribution ofprobabilities differs from the independence distribution can be used asa measure of the statistical dependence of the random events.

[0220] Information theoretic entropy can provide a convenient metric forestimating the difference between distributions. The entropy, H(X)(measured in bits) of the distribution of a discrete random variable Xwith n states can be $\begin{matrix}{{H(X)} = {- {\sum\limits_{l = 1}^{n}\quad {p_{1}\log_{2}p_{1}}}}} & (2)\end{matrix}$

[0221] where p₁ is the probability of state i. Entropy can be maximizedwhen a distribution is uniform. For example, FIG. 17 shows a graph 1702of the entropies 1722 of a single variable, discrete 2-statedistributions and how their probabilities vary.

[0222] Similarly, for a multivariate distribution constrained byspecified marginal distributions, the distribution that maximizesentropy can be the independence distribution. Therefore, given a jointdistribution with fixed marginals, the distribution that minimizesentropy can be the distribution for which the variables are completelydependent. Dependence can be viewed as constraint between variables andas constraint is reduced, entropy increases. Information theoreticanalysis of a distribution is then the measurement of constraint.Decreasing entropy can indicate dependence (minimal entropy, maximumconstraint), and increasing entropy can indicate independence (maximumentropy, minimum constraint). Assuming some constraint betweenvariables, sampled distribution can lie somewhere between completedependence and independence and have a measurable entropy.

[0223] If we are analyzing the joint distribution of the variables X andY, the entropy for this sampled distribution can be H(XY). The entropiesof the variables X and Y measured separately are H(X) and H(Y) and canbe computed using the marginals of the joint distribution.

[0224] Since H(X) and H(Y) are calculated from the marginals and entropycan be logarithmic,

H(X)+H(Y)=H(XY)   (3)

[0225] if there is no constraint between X and Y.

[0226] Or:

H(XY)=H(X)+H(Y)   (4)

[0227] if and only if X and Y are independent.

[0228] This equality can indicate that there is no relationship betweenX and Y and the joint distribution of the variables is the independencedistribution.

[0229] Information transmission T can be the measure of the distancebetween distributions along the continuum described above. For discreterandom variables X and Y, T(X:Y) the information transmission between Xand Y, is computed:

T(X:Y)=H(X)+H(Y)−H(XY)   (5)

[0230] T(X:Y) is the difference between the entropies of theindependence distribution and the sampled joint distribution. The degreeof dependence between X and Y can therefore be computed by measuringinformation transmission. A small value for T(X:Y) indicates thevariables X and Y are nearly independent, whereas a large value suggestsa high degree of interaction.

[0231] In a directed system, such as a predictive model, the measure ofinformation transmission between the distribution of an independentvariable X and a dependent variable Y can be used to gauge thepredictive value of X. The goal can be to find a subset S of theindependent variables V such that, for the set of dependent variables D:

T(D:V)≈T(D:S)   (6)

[0232] However, as discussed, the modeling technique to be employed maylimit the cardinality of S so the filtering process can be guided by thefollowing considerations:

[0233] 1. if S′ is any subset of V smaller than S, then T(D:S′) issignificantly smaller than T(D:S).

[0234] 2. if S′ is any subset of V larger than S, then T(D:S′) is notsignificantly larger than T(D:S)

[0235] Since information theoretic transmission can measure the degreeof difference between distributions of variables, without regard to thenature of the difference, the technique can be considered “model free”.This property allows the methodology to work as an effective filterregardless of the subsequent modeling techniques employed.

[0236] When this type of feature selection was applied to tenureprediction, 56 questions (see Tables 4 and 5) were selected has havingthe most predictive value with respect to applicant tenure.

[0237] Once the set of predictor variables or inputs has been definedand the output criterion variable specified, a neural network model canbe trained. For the tenure prediction model, 2084 cases were available.This sample was divided into training, test and verification sets. Thetraining set contained 1784 cases and the verification and test setscontained 150 cases each.

[0238] The best performing neural network architecture was found to be asingle hidden layer feed-forward network with 56 input nodes and 40hidden layer nodes.

[0239] The network was developed with the STATISTICA Neural Networkpackage using a combination of quick-propagation and conjugate gradienttraining.

[0240] The performance on the training and verification sets began todiverge significantly after 300 epochs. This was deemed to be the pointof over-training. Optimal performance on the hold-out sets was achievedat 100 epochs. The results are shown in Table 3, which contains finaldistribution statistics of model output for each of the three datasubsets. Unadjusted correlation and significance statistics are inrelation to actual tenure. By any standard, an employee selectionprocedure with a correlation in the 0.5 range with a job-relatedcriteria is not merely acceptable, but exceptional. Many validatedselection procedures in use today were implemented on the basis ofvalidity coefficients in the range of 0.2 to 0.3. TABLE 3 SummaryStatistics of Model Output Train Verify Test Data Mean 73.42657 82.8933371.03333 Data S.D. 70.92945 71.22581 62.16501 Error Mean −0.4771 −7.25827.440303 Error S.D. 60.84374 60.93211 53.80157 Correlation 0.5143490.51901 0.503975 Significance 0.000 0.000 0.000

[0241] Based on the correlation between prediction and the hold-outsets, the expected correlation between predictive model output andactual tenure for future applicants should be in the range of 0.5.

[0242] As described in the example, information theoretic featureselection was used to identify fifty-six biodata and personalityassessment item responses that were related to employee tenure in asample of over two thousand employees at a national video rental chain.The data was collected via interactive electronic survey administrationon a network of screen phones deployed in many regions of the U.S.

[0243] A fully-connected, feed-forward backpropagation neural networkwas trained to produce an estimate of tenure in days using thesefifty-six predictor variables (e.g., answers to the questions) asinputs. Network architecture consisted of 56 input neurons or nodes, ahidden layer of forty nodes and one output node. Conjugate gradientdescent training resulted in convergence between training and test setminimum error in about 300 iterative training exposures to the data.Model performance on an independent hold-out sample obtained astatistically significant correlation of 0.5 with actual tenure. Theseresults are well within the range of acceptable performance for acriterion-referenced employee selection procedure and represent asignificant improvement over many systems.

[0244] In the example, based on information theoretic analysis, theresponses to the questions shown in Tables 4 and 5 were deemed to be themost predictive. The following descriptions are the questions in theirentirety accompanied by the possible responses.

[0245] To determine that these questions were the most predictive,information theoretic analysis of the joint distribution of the response(alone or together with other responses) and the dependent variable,tenure, was performed. The nature of the relationship between a specificresponse and the Criterion variables may not be known, however thepredictive success of the neural model suggests this relationship has,to some degree, been encoded in the weight matrix of the neural network.TABLE 4 Pre-hire Content Examples  1. How long do you plan to stay withthis job if hired? 1 - Less than 6 months 2 - 6-12 months 3 - More than1 year  2. Have you ever worked for this employer before? 1 - Yes 2 - No 3. Reason for leaving? (if previously employed by this employer)  4.Which type of position do you desire? 1 - Store Director 2 - AssistantDirector 3 - Customer Service Representative 4 - Shift Leader 5 - Let'sDiscuss  5. What do you expect to earn on an hourly basis? (hourly wagegiven)  6. Desired Schedule? 1 - Regular (not seasonal) 2 - Seasonal  7.Desired Hours? 1 - Full time 2 - Part time  8. When would you beavailable to start? 1 - Right Away (within the next day) 2 - SpecificDate (if not available to start within the next day)  9. HighestEducation Level? 1 - 2 Years of College or Less: 1 - Not indicated 2 -Less than HS Graduate 3 - HS graduate or equivalent 4 - Some college 5 -Technical School 6 - 2-year college degree 2 - More than 2 years ofcollege 1 - Bachelor's level degree 2 - Some graduate school 3 -Master's level degree 4 - Doctorate (academic) 5 - Doctorate(professional) 6 - Post-doctorate 7 - Degree not completed 8 - 2-yearcollege degree 10. What was your reason for leaving? (last job) 1 -Voluntarily quit 2 - Involuntarily terminated 3 - Laid off 4 - Stillthere 11. What was/is your job title? (last job) 1 - Cashier 2 - Stockperson 3 - Customer Service Representative 4 - Management 5 - Other 12.Please describe the area you worked in. (last job) 1 - Apparel 2 -Inventory 3 - Customer service 4 - Food service 5 - Operations 6 -Computers/Electronics 7 - Merchandising 8 - Personnel 9 - Other 13. Whatwas/is you supervisor's last name? (given or not given) 14. May wecontact this employer? 1 - Yes 2 - No 15. What was your reason forleaving? (prior job) 1 - Voluntarily quit 2 - Involuntarily terminated3 - Laid off 4 - Still there 16. What was/is your job title? (prior job)1 - Cashier 2 - Stock person 3 - Customer Service Representative 4 -Management 5 - Other 17. Please describe the area you worked in. (priorjob) 1 - Apparel 2 - Inventory 3 - Customer service 4 - Food service 5 -Operations 6 - Computers/Electronics 7 - Merchandising 8 - Personnel 9 -Other 18. What was/is you supervisor's last name? (prior job) (given ornot given) 19. May we contact this employer? (prior job) 1 - Yes 2 - No20. What was your reason for leaving? (prior to prior job) 1 -Voluntarily quit 2 - Involuntarily terminated 3 - Laid off 4 - Stillthere 21. What was/is your job title? (prior to prior job) 1 - Cashier2 - Stock person 3 - Customer Service Representative 4 - Management 5 -Other 22. Please describe the area you worked in. (prior to prior job)1 - Apparel 2 - Inventory 3 - Customer service 4 - Food service 5 -Operations 6 - Computers/Electronics 7 - Merchandising 8 - Personnel 9 -Other 23. What was/is you supervisor's last name? (prior to prior job)(given or not given) 24. May we contact this employer? (prior to priorjob) 1 - Yes 2 - No 25. Academic Recognitions? (listed or not listed)26. Other Recognitions? (listed or not listed) 27. Have you previouslyapplied for employment at this employer? 1 - Yes 2 - No 28. ReferralSource 1 - Referred to this employer by Individual or Company 1 - Agency2 - Client Referral 3 - College Recruiting 4 - Employee Referral 5 -Former Employee 6 - Executive Referral 7 - Executive Search 2 - OtherSource of Referral 1 - Advertisement 2 - Job Fair 3 - Job Posting 4 -Open House 5 - Other Source 6 - Phone Inquiry 7 - Unknown 8 -Unsolicited 9 - Walk In 29. Last name of referral (listed or not listed)30. Any other commitments? (listed or not listed) 31. Any personalcommitments? (listed or not listed)

[0246] The possible responses to the question of Table 5 are as follows:“1—It is definitely false or I strongly disagree, 2—It is false or Idisagree, 3—It is true or I agree, 4—It is definitely true or I stronglyagree.” TABLE 5 Pre-hire Content Examples (e.g., Hourly Workers)  1. Youhave confidence in yourself.  2. You are always cheerful.  3. You getmad at yourself when you make mistakes.  4. You would rather work on ateam than by yourself.  5. You try to sense what others are thinking andfeeling.  6. You can wait patiently for a long time.  7. When someonetreats you badly, you ignore it.  8. It is easy for you to feel whatothers are feeling.  9. You keep calm when under stress. 10. You like tobe alone. 11. You like to talk a lot. 12. You don't care what peoplethink of you. 13. You love to listen to people talk about themselves.14. You always try not to hurt people's feelings. 15. There are somepeople you really can't stand. 16. People who talk all the time areannoying. 17. You are unsure of yourself with new people 18. Slow peoplemake you impatient. 19. Other people's feelings are their own business.20. You change from feeling happy to sad without any reason. 21. Youcriticize people when they deserve it. 22. You ignore people you don'tlike. 23. You have no big worries. 24. When people make mistakes, youcorrect them. 25. You could not deal with difficult people all day.

EXAMPLE 36 Exemplary Implementation Using

[0247] Information-Theoretic Feature Selection Information-theoreticfeature selection can be used to choose appropriate inputs for a model.In the following example, the source for the data used to develop themodel was a large national video rental company. The sample containsover 2000 cases, with 160 responses to application questions collectedprior to hiring and tenure (in days) for former employees. The model wasconstructed to predict the length of employment for a given applicant,if hired.

[0248] The application itself consists of 77 bio-data questions (e.g.,general, work related, information, job history, education and referralsquestions) and 83 psychometric questions. The psychometric assessmentportion was designed to predict the reliability of an applicant in anhourly, customer service position. For the purposes of modeldevelopment, each question response was treated as a single feature andthe reliability score was not provided to the neural network or featureselection process. While any information gathered during the applicationprocess may have predictive value, the set of input variables(independent variables or “IVs”) can be reduced. Possible justificationsare as follows:

[0249] 1. Not all potential IVs may have significant predictive value.The use of variables with little or no predictive value as inputs canadd noise. Adding IVs to the model which cannot improve predictivecapability may degrade prediction since the network may need to adapt tofilter these inputs. This can result in additional training time andneural resources.

[0250] 2. Predictive models can provide a mapping from an input space toan output space. The dimensionality of this input space increases withthe number of inputs. Thus, there are more parameters required to coverthe mapping which in turn increases the variance of the model (in termsof the bias/variance dilemma); such a problem is sometimes referred toas the “curse of dimensionality.”

[0251] IVs with less predictive power can be eliminated in favor of aless complex neural network model by applying feature selection. Suchmethods fall into two general categories: filters and wrappers, eitherof which can be used.

[0252] 1. Wrappers can use the relationship between model performanceand IVs directly by iteratively experimenting with IV subsets. Since thenature of the bias of the feature selection method matches that of themodeling technique, this approach can be theoretically optimal if thesearch is exhaustive.

[0253] The exhaustive application of wrappers can be computationallyoverwhelming for most modeling problems since the number of possiblesubsets is $\begin{matrix}{\begin{pmatrix}n \\k\end{pmatrix} = \frac{n!}{{k!}{\left( {n - k} \right)!}}} & (7)\end{matrix}$

[0254] where n is the total number of IVs and k is the cardinality ofthe subset of features.

[0255] Additionally, there can be non-determinism within the modelingprocess. In neural modeling, though training algorithms are typicallydeterministic, random initialization of the weight parameters varies theresults of models developed with the same inputs. Therefore, evenexhaustive trials may not prove conclusive with respect to estimatingthe predictive value of a set of features.

[0256] 2. Filters can analyze the relationship between sets of IVs anddependent variables (DVs) using methods independent of those used todevelop the model.

[0257] The bias of the filter may be incompatible with that of themodeling technique. For example, a filter may fail to detect certainclasses of constraint, which the subsequent modeling stage may utilize.Conversely, the filter may identify relations which cannot besuccessfully modeled. Ideally, a filter can be completely inclusive inthat no constraint which might be replicated by the subsequent modelingstage would be discarded.

[0258] Information-theoretic feature selection can make use of thestatistical theory of independent events. Events P₁, P₂, . . . , P_(n)are considered statistically independent if and only if the probabilityP, that they all occur on a given trial is $\begin{matrix}{P = {\prod\limits_{l = 1}^{n}\quad p_{1}}} & (8)\end{matrix}$

[0259] The degree to which a joint distribution of probabilitiesdiverges from the independence distribution may be used as a measure ofthe statistical dependence of the events.

[0260] Information-theoretic entropy can provide a convenient metric forquantifying the difference between distributions. The entropy, H(X)(measured in bits), of the distribution of a discrete random variable,X, with n states can be $\begin{matrix}{{H(X)} = {- {\sum\limits_{l = 1}^{n}\quad {p_{i}\log_{2}p_{i}}}}} & (9)\end{matrix}$

[0261] where p_(i) is the probability state i.

[0262] Entropy can be maximized when a distribution is most uncertain.If a distribution is discrete, this occurs when it is uniform. FIG. 17shows a graph of the entropies of a single variable, 2-statedistribution as the state probabilities vary.

[0263] For a multivariate distribution constrained by fixed marginals,the distribution which maximizes entropy can be the independencedistribution (calculated as the product of the marginals). Thedistribution which minimizes entropy can be the distribution for whichthe variables are completely dependent.

[0264] Dependence can be constraint between variables, so as constraintis reduced, entropy increases. Information-theoretic analysis cantherefore be used to measure constraint. For a joint distribution ofdiscrete variables, X and Y, the total entropy, H(XY) can be$\begin{matrix}{{H({XY})} = {- {\sum\limits_{i,j}\quad {p_{ij}\log_{2}p_{ij}}}}} & (10)\end{matrix}$

[0265] where p_(ij) is the probability of state i,j occurring in thejoint distribution of X and Y, where i designates the state of X and jis the state of Y. The entropies of X and Y are computed with themarginals of the joint distribution $\begin{matrix}{{H(X)} = {- {\sum\limits_{i}\quad {\left( {\sum\limits_{j}\quad p_{ij}} \right)\log_{2}\quad \left( {\sum\limits_{j}\quad p_{ij}} \right)}}}} & (11) \\{{H(Y)} = {- {\sum\limits_{j}\quad {\left( {\sum\limits_{i}\quad p_{ij}} \right)\log_{2}\quad \left( {\sum\limits_{i}\quad p_{ij}} \right)}}}} & (12)\end{matrix}$

[0266] Information transmission (or “mutual information”) can be themeasure of the distance between the independence and observeddistributions along the continuum discussed above. For X and Y, T(X:Y)(the information transmission between X and Y), is computed

T(X:Y)=H(X)+H(Y)−H(XY)   (13)

[0267] In a directed system, the measure of information transmissionbetween the distribution of an independent variable X and a dependentvariable Y is a gauge of the predictive value of X. H(X)+H(Y)=H(XY) ifand only if there is no constraint between X and Y, in which case Xwould be a poor predictor for Y.

[0268] In order for a computed transmission value, T, to be consideredan accurate measure of existing constraint, the statistical significanceof T for some confidence level, α, can be determined using the χ² test.The degrees of freedom (df) for a transmission, T(X:Y), can becalculated

df _(T(XY)) =df _(XY) −dF _(X) −df _(Y)   (14)

[0269] As the size of the joint distribution increases, so does the dffor the significance of the transmission value. Since χ² significancedecreases as df increases, the data requirements for transmissionscontaining a large number of variables can quickly become overwhelming.

[0270] A superior feature set can be determined. A goal can be todiscover a subset S of the independent variables V that has the samepredictive power as the entire set with respect to the dependentvariables, D.

T(V:D)≈T(S:D)   (15)

[0271] The filtering process can therefore be guided by the following:

[0272] 1. if S′ is any subset of V smaller than S, then T(S′:D) issignificantly smaller than T(S:D).

[0273] 2. if S′ is any subset of V larger than S, then T(S′:D) is notsignificantly larger than T(S:D).

[0274] Higher-order interactions are synergies between variables wherethe predictive power of a set of variables is significantly higher thanthat of the sum of the individual variables. In terms of informationtransmission for the IVs X₁, . . . , X_(n), and dependent variable D,this is represented,

T(X ₁ :D)+ . . . +T(X _(n) :D)<T(X ₁ , . . . , X _(n) :D)   (16)

[0275] An illustration of this phenomenon among discrete binaryvariables: A, B and C, is shown by the contingency table in Tables 6Aand 6B. TABLE 6A Contingency Table for Distribution ABC, C = 0 B = 0 B =1 A = 0 1/4 0 A = 1 0 1/4

[0276] TABLE 6B Contingency Table for Distribution ABC, C = 1 B = 0 B =1 A = 0 0 1/4 A = 1 1/4 0

[0277] For the illustrated system, the following transmissions arecomputed:

[0278]T(A:C)=H(A)+H(C)−H(AC)=0 bits

T(B:C)=H(B)+H(C)−H(BC)=0 bits

T(AB:C)=H(AB)+H(C)−H(ABC)=1 bit

[0279] Knowledge of A or B individually does not reduce the uncertaintyof C, but knowledge of A and B eliminates uncertainty since only onestate of C is possible. With only first order transmissions values, Aand B would not appear to be predictive features, when in fact, togetherthey are ideal.

[0280] Higher order interactions were observed in the video clerk tenuredata. Table 7 lists the top ten single variable transmissions betweenthe psychometric questions and tenure. Table 8 shows the top five, twoand three variable transmissions. Each of the most predictive sets ofquestions (based on transmission values) in both the second and thirdorder lists, T(q35 q73:tenure) and T(q4 q12 q39:tenure), contain onlyone question from the top ten most predictive questions based on firstorder transmissions. TABLE 7 Single Order Transmissions BetweenPsychometrics and Tenure variables trans. % H(DV) df χ²sig.T(q83:tenure) 0.0168 0.754 27 0.999 T(q3:tenure) 0.0140 0.628 27 0.991T(q63:tenure) 0.0135 0.607 27 0.987 T(q65:tenure) 0.0133 0.598 27 0.985T(q48:tenure) 0.0133 0.595 27 0.984 T(q44:tenure) 0.0132 0.593 27 0.984T(q35:tenure) 0.0128 0.573 27 0.977 T(q21:tenure) 0.0127 0.569 27 0.975T(q8:tenure) 0.0123 0.553 27 0.967 T(q69:tenure) 0.0123 0.552 27 0.966

[0281] TABLE 8 Higher (second and third) Order Transmissions betweenPsychometrics and Tenure variables trans. % H(DV) df χ²sig. T(q35q73:tenure) 0.0593 2.663 135 1.00 T(q21 q83:tenure) 0.0588 2.639 1351.00 T(q39 q65:tenure) 0.0585 2.627 135 1.00 T(q61 q70:tenure) 0.05692.553 135 0.999 T(q44 q53:tenure) 0.0567 2.546 135 0.999 T(q4 q12q39:tenure) 0.1808 8.112 567 0.921 T(q10 q39 q65:tenure) 0.1753 7.864567 0.811 T(q4 q39 q44:tenure) 0.1720 7.718 567 0.712 T(q4 q39q51:tenure) 0.1718 7.709 567 0.705 T(q52 q61 q70:tenure) 0.1717 7.702567 0.700

[0282] Such interactions can complicate the search for the optimal set Ssince the members of V may not appear as powerful predictors incalculated transmissions using sets of features of cardinality less than|S| (the cardinality of the optimal subset S).

[0283] Due to issues of χ² significance, it is frequently overwhelmingto calculate significant transmission values for sets of variables ofcardinality approaching |S|. Additionally, since the number of subsetsof a given cardinality soon become very large, even if the significanceissues were addressed, computational limitations would persist.

[0284] In feature selection algorithms that approximate an exhaustivesearch for S by computing only pairwise transmissions, higher-orderinteraction effects are not detected. Such methods may not accuratelyapproximate S since only variables which are strong single variablepredictors will be selected.

[0285] Based on the following guidelines, heuristics were applied in aneffort to address the problems of combinatorics and significance inmeasuring higher-order relations.

[0286] Although it is possible for members of the optimal subset of IVs,S, to be completely absent from all large lower order transmissions,this is probably unlikely. An omission can be increasingly unlikely asthe order of the transmissions calculated approaches |S|. It istherefore likely that significant members of S will appear in the top ntransmissions of the highest order transmission computed, where n issufficiently large. Thus, as n→|S|, the union of the set of IVsappearing in the most predictive transmissions will probably approach S.

[0287] With these guidelines, a process for generating an approximationto S (S′) given the set V of significant IVs and the set D of all DVs,can be presented.

[0288] In the following process (1-6), T_(k) will be used to denote theset of transmissions of order k (containing k IVs) from a set of nfeatures.

[0289] 1. Calculate the transmissions, T_(k) for the highest order, k,for which the $\left( \frac{n}{k} \right)$

[0290] transmissions may be calculated.

[0291] 2. Choose the m unique transmissions of the greatest magnitudefrom T_(k) to be the base set for higher-order transmissions.

[0292] 3. Generate T′_(k+1) by adding the IV to numbers of T_(k) whichgenerates the set T_(k+1) with the largest transmission values. Notethat T′_(k+1) is a subset of T_(k+1) since it contains only thosemembers of T_(k+1) which can be generated from T_(k) by adding oneindependent variable to each transmission.

[0293] 4. Discard any duplicate transmissions.

[0294] 5. Repeat Steps 3 and 4 until χ² significance is exhausted.

[0295] 6. Take the union of the variables appearing in as many of themost predictive transmissions as is necessary to generate a set of size|S|. This union is S′, the approximation of the set S.

[0296] Since |S| is unknown, this value is estimated. However,0≦|S|≦|V|, so it is often feasible to experiment with the S′ for eachcardinality.

[0297] An issue raised by feature selection processes is the effect ofdependence between members of S′. This dependence may be viewed as theredundancy in the predictive content of the variables. One solutionproposed is to calculate the pairwise transmissions T(s′_(i): s′_(j)),between features s′_(i) and s′_(j), from a candidate S′. Features whichexhibit high dependence (high pairwise transmissions) are penalized withrespect to the likelihood of their inclusion in the final S′.

[0298] Dependence between features is dealt with implicitly in theprocess above since such dependence will reduce the entropy, therebyreducing the magnitude of the transmission between a set of features andthe set of dependent variables. Highly redundant feature sets will havelow transmission values relative to less redundant sets of the samecardinality and will therefore be less likely to contribute to S′.

[0299] While tenure in days is a discrete measure, the number ofpossible states makes it difficult to use the variable withouttransformation since a large number of states makes the jointdistribution sparse (high df relative to the data population) and anytransmissions calculated statistically insignificant. Since tenure is anordered variable, applying a clustering algorithm was not problematic.

[0300] Clustering is a form of compression, so care can be taken tominimize information loss. The clustering phase was guided by efforts tomaximize the entropy of the clustered variable within the confines ofthe needs of statistical significance.

[0301] Though transmission values did vary across clustering algorithmsand granularity, the results in terms of S′ were consistent.

[0302] Transmissions were calculated by combining cluster analysis andinformation-theoretic analysis. For the video clerk data set (containing160 IVs) it was decided that the cardinality of the sets of IVs forwhich transmissions could be calculated was 4. From there, twoadditional orders of cardinality were calculated by supplementing the4th order transmissions (as described in step 3 of the process). Theunion of independent variables appearing in the largest transmissionswas taken to be S′. Experimentation with neural models using S′ ofdifferent cardinalities yielded the best results when |S′|=56.

[0303] An interesting aspect of the application questions chosen by thefeature selection method was the mix of bio-data and psychometrics. Ofthe 56 features used as inputs for the most successful model, 31 camefrom the bio-data section of the application and 25 came from thepsychological assessment. Of particular interest was the “coupling” ofcertain bio-data and assessment questions. Such pairs would appeartogether throughout the analysis of transmission over a range ofcardinalities. (e.g., they would appear as a highly predictive pair andwould subsequently appear together in higher-order sets of IVs).

[0304] The synergistic effect between the two classes of question becameapparent when models were generated using exclusively one class or theother (using only psychometrics or only bio-data questions). Withcomparable numbers of inputs, these models performed significantly worsethan their more diverse counterparts. These results are particularlyinteresting since psychological assessments typically do not includeresponses from such diverse classes of questions.

[0305] In the example, the most successful neural model developed was asingle hidden layer, feed-forward neural network with 56 inputs(|S′|=56), slid 40 hidden nodes. The network was trained using theconjugate gradient method. Of the total data set size of 2084, 1784 wereallocated to the training set and 300 were “hold-out”.

[0306] The performance measures of behavioral prediction models can bemeasured using the correlation coefficient. For the neural modeldescribed, the correlation between prediction and actual tenure for thehold-out sample was p=0.51. For comparison, a number of other modelswere generated using either no feature selection or alternate featureselection methods. These models used the same network architecture andtraining algorithm. The best model generated using the entire data set(e.g., all features), was a 160-90-1 configuration (160 inputs and 90hidden layer nodes) which achieved a maximum hold-out correlation ofp=0.44. Alternate feature selection algorithms: genetic algorithms, andforward and reverse stepwise regression, using the same number offeatures (56), failed to achieve a hold-out correlation better thanp=0.47.

[0307] Information-theoretic feature selection is a viable and accuratemethod of identifying predictors of job performance in employeeselection. The capacity to identify non-linear and higher-orderinteractions ignored by other feature selection methods represents asignificant technique in constructing predictive models.

Alternatives

[0308] It should be understood that the programs, processes, or methodsdescribed herein are not related or limited to any particular type ofcomputer apparatus, unless indicated otherwise. Various types of generalpurpose or specialized computer apparatus may be used with or performoperations in accordance with the teachings described herein. Elementsof the illustrated embodiment shown in software may be implemented inhardware and vice versa. In view of the many possible embodiments towhich the principles of our invention may be applied, it should berecognized that the detailed embodiments are illustrative only andshould not be taken as limiting the scope of our invention. Rather, weclaim as our invention all such embodiments as may come within the scopeand spirit of the following claims and equivalents thereto.

We claim:
 1. An apparatus for assisting in determining the suitabilityof an individual for employment by an employer, the apparatuscomprising: an electronic data interrogator operable to present a firstset of a plurality of questions to the individual; an electronic answercapturer operable to electronically store the individual's responses toat least a selected plurality of the first set of questions presented tothe individual; an electronic predictor responsive to the stored answersand operable to predict at least one post-hire outcome if the individualwere to be employed by the employer, the predictor providing aprediction of the outcome based upon correlations of the stored answerswith answers to sets of questions by other individuals for whichpost-hire information has been collected; and an electronic resultsprovider providing an output indicative of the outcome to assist indetermining the suitability of the individual for employment by theemployer.
 2. An apparatus according to claim 1 wherein the post-hireoutcome indicates whether the individual is predicted to be eligible forre-hire after termination.
 3. An apparatus according to claim 1 whereinthe post-hire outcome indicates whether the individual is predicted tobe involuntarily terminated.
 4. An apparatus according to claim 1wherein the post-hire outcomes indicate whether the individual ispredicted to be involuntarily terminated and whether the individual ispredicted to be eligible for re-hire after termination.
 5. An apparatusaccording to claim 1 wherein at least one of the predicted outcomes is apredicted probability that a particular outcome value range will beobserved.
 6. An apparatus according to claim 1 wherein at least one ofthe predicted outcomes is a predicted value for a continuous variable.7. An apparatus according to claim 1 wherein the predicted outcome is apredicted range of values for a continuous variable.
 8. An apparatusaccording to claim 1 wherein the predicted outcome indicates whether theindividual will belong to a particular group.
 9. An apparatus accordingto claim 1 wherein at least one of the predicted outcomes is a predictedranking of the individual for the outcome.
 10. An apparatus according toclaim 1 wherein at least one of the predicted outcomes indicates apredicted employment tenure for the individual.
 11. An apparatusaccording to claim 1 wherein at least one of the predicted outcomesindicates a predicted number of accidents for the individual.
 12. Anapparatus according to claim 1 wherein at least one of the predictedoutcomes indicates a predicted sales level for the individual.
 13. Anapparatus according to claim 1 wherein the predictor comprises anartificial intelligence-based prediction system.
 14. An apparatusaccording to claim 1 wherein the data interrogator is located at a firstlocation and the predictor is located at a second location which isremote from the first location.
 15. An apparatus according to claim 14wherein the data interrogator and the predictor are selectivelyelectronically interconnected through a network.
 16. An apparatusaccording to claim 15 wherein the network is the worldwide web.
 17. Anapparatus according to claim 15 wherein the network is a telephonenetwork.
 18. An apparatus according to claim 15 wherein the network is asatellite network.
 19. An apparatus according to claim 1 wherein thefirst set of questions may be varied.
 20. An apparatus according toclaim 19 wherein the predictor is operable to determine and indicate alack of a correlation between one or more questions of the first set ofquestions and at least one of the predicted outcomes, whereby questionswhich lack the correlation may be discarded or modified.
 21. Anapparatus according to claim 1 wherein at least one of the predictedoutcomes is longevity with an employer and the answers to sets ofquestions by other individuals comprise answers by employees of theemployer for whom longevity has been determined.
 22. An apparatusaccording to claim 1 in which the predictor comprises at least one modelwhich provides a predictor of the probability of the individualexhibiting at least one of the predicted outcomes, the model being basedon correlations between the at least one of the predicted outcomes andthe answers to questions by the other individuals, including answers byat least some employees of the employer, the model taking at leastselected answers of the stored answers as inputs to the model, aprobability of the individual exhibiting the at least one of thepredicted outcomes being provided as an output of the model.
 23. Anapparatus according to claim 22 wherein the model comprises at least oneneural network.
 24. An apparatus according to claim 1 wherein thepredictor is responsive to the stored answers and operable to predictplural outcomes if the individual were to be employed by the employer.25. A method for assessing suitability of persons for employment basedon information for hired employees, the method comprising: collectingpre-hire applicant information for hired employees before they arehired; collecting post-hire measures of the job effectiveness of hiredemployees; constructing an artificial intelligence model identifyingassociations of patterns within the pre-hire data associated withpatterns of job effectiveness in the post-hire data; collecting pre-hireinformation for a new applicant; and applying the artificialintelligence model to the pre-hire information for the new applicant toprovide a prediction of the new applicant's suitability for employment.26. The method of claim 25 further comprising: collecting post-hireinformation for the new applicant; and using at least the pre-hire andpost-hire information for the new applicant to refine the artificialintelligence model.
 27. The method of claim 25 further comprising:constructing at least one other artificial intelligence model of adifferent type; and assessing the relative effectiveness of theartificial intelligence models at predicting suitability of employeesfor employment based on actual employment effectiveness of employeeshired based on the models.
 28. An apparatus for assisting in determiningthe suitability of an individual for employment by an employer, theapparatus comprising: means for electronically presenting a first set ofa plurality of questions to the individual; means for electronicallystoring the individual's responses to at least a selected plurality ofthe first set of questions presented to the individual; responsive tothe stored answers, means for predicting at least one post-hire outcomeif the individual were to be employed by the employer, the means forpredicting providing a prediction of the outcome based upon correlationsof the at least one characteristic with answers to sets of questions byother individuals and the closeness of the stored answers to suchcorrelations; and means for providing an output indicative of theoutcome to assist in determining the suitability of the individual foremployment by the employer.
 29. An artificial intelligence-based systemfor predicting employee behaviors based on pre-hire informationcollected for the employee, the system comprising: an electronic devicefor presenting an employment application comprising a set of questionsto an employment candidate, wherein the electronic device is operable totransmit answers of the employment candidate to a central store ofemployee information, wherein the central store of employee informationcomprises information collected for a plurality of candidate employeesand a plurality of hired employees; an artificial intelligence-basedmodel constructed from information collected from the hired employeesbased on answers provided by the hired employees and employmentbehaviors observed for the hired employees; a software system forsupplying the answers of the employment candidate to the artificialintelligence-based model to produce predicted employment behaviors forthe employment candidate; and a report generator to produce a hiringrecommendation report for the employment candidate based on thepredicted employment behaviors of the employment candidate.
 30. Acomputer-implemented method of predicting employment performancecharacteristics for a candidate employee based on pre-hire informationcollected for hired employees, the method comprising: collecting dataindicating pre-hire information for a plurality of the hired employees;collecting data indicating post-hire outcomes for the hired employees;constructing an artificial intelligence-based model from the pre-hireinformation and the post-hire outcomes for the employees; from thecandidate employee, electronically collecting data indicating pre-hireinformation of the candidate employee; and applying the model to thecollected pre-hire information of the candidate employee to generate oneor more predicted post-hire outcomes for the candidate employee.
 31. Themethod of claim 30 wherein collecting data from the candidate employeecomprises electronically presenting a set of questions at an electronicdevice and electronically collecting answers to the questions at theelectronic device.
 32. The method of claim 30 wherein the pre-hireinformation comprises one or more pre-hire characteristics andconstructing the model comprises: identifying one or more pre-hirecharacteristics as ineffective predictors; and responsive to identifyingthe pre-hire characteristics as ineffective predictors, omitting theineffective predictors from the model.
 33. The method of claim 30further comprising: providing a report indicating applicant flow. 34.The method of claim 30 wherein constructing the model comprises:constructing a plurality of proposed models, wherein at least two of themodels are of different types; and selecting a superior proposed modelas the model to be used.
 35. The method of claim 34 wherein at least twoof the proposed models are different neural network types.
 36. Themethod of claim 35 wherein the two proposed models are both feed-forwardneural networks.
 37. The method of claim 35 wherein the two proposedmodels are chosen from the following: back propagation, conjugategradients, quasi-Newton, Levenberg-Marquardt, quick propagation,delta-bar-delta, linear, radial basis function, and generalizedregression network.
 38. The method of claim 30 wherein at least one ofthe predicted post-hire outcomes is denoted as a probability that aparticular value range of a job effective measure will be observed for acandidate employee.
 39. The method of claim 30 wherein at least one ofthe predicted post-hire outcomes is denoted as a value for a continuousvariable.
 40. The method of claim 30 wherein at least one of thepredicted post-hire outcomes is denoted as a relative ranking for anoutcome.
 41. The method of claim 40 wherein the ranking is relative toother employment candidates.
 42. The method of claim 40 wherein theranking is relative to the hired employees.
 43. The method of claim 30further comprising: storing a relative importance of one or moreparticular post-hire outcomes; and generating automated hiringrecommendations based on the predicted post-hire outcomes for thecandidate employees and the importance of the post-hire outcomes. 44.The method of claim 30 further comprising: refining the model based onnewly-observed post-hire outcomes.
 45. The method of claim 30 whereinthe pre-hire information comprises answers to questions on a jobapplication, the method further comprising: identifying one or morequestions as ineffective predictors; responsive to identifying thequestions as ineffective predictors, modifying the job application byremoving the questions; collecting new pre-hire information foradditional candidate employees based on the modified job application;collecting new post-hire information for the additional candidateemployees; and constructing a refined artificial-intelligence modelbased on the additional pre-hire and post-hire information for theadditional candidate employees.
 46. The method of claim 45 furthercomprising: responsive to determining pre-hire and post-hire informationhas been collected for a sufficient number of additional employees,providing an indication that a refined model can be constructed.
 47. Themethod of claim 45 further comprising: providing a report indicating theidentified questions are ineffective predictors.
 48. The method of claim45 further comprising: adding one or more new questions to the modifiedjob application before collecting additional pre-hire information. 49.The method of claim 48 wherein the new questions are composed based onjob skills appropriate for a particular job related to the jobapplication.
 50. The method of claim 48 further comprising: evaluatingthe effectiveness of the new questions.
 51. An artificialintelligence-based employee performance prediction system comprising: aset of pre-hire characteristic identifiers; a set of post-hire outcomeidentifiers; a collection of data for employees, wherein the dataincludes values associated with the pre-hire identifiers and thepost-hire identifiers; and an artificial intelligence-based model chosenfrom a set of candidate models, the artificial intelligence-based modelexhibiting superior ability at predicting values associated with thepost-hire outcome identifiers based on values associated with thepre-hire characteristic identifiers in comparison to the other candidatemodels.
 52. A computer-readable medium having a collection ofemployment-related data, the data comprising: pre-hire information for aplurality of employees, wherein the pre-hire information comprisesinformation electronically-collected from an applicant, wherein theinformation comprises a plurality of pre-hire characteristics; post-hireinformation for at least some of the plurality of employees, wherein theinformation comprises a plurality of post-hire outcomes; and a datastructure identifying which of the pre-hire characteristics areeffective in predicting a set of one or more of the post-hire outcomesfor a job applicant.
 53. A method for providing an automated hiringrecommendation for a new potential employee, the method comprising:collecting pre-hire information for potential employees; storing thepre-hire information for the potential employees in a database; afterhiring a plurality of the potential employees, collecting employmentperformance information for at least some of the hired employees;storing the employment performance information collected from the hiredemployees; constructing an artificial intelligence-based model based oncorrelations between the pre-hire information and the employmentperformance information collected from one or more of the hiredemployees; collecting pre-hire information for a new potential employee;based on the artificial intelligence-based model, providing an automatedhiring recommendation for the new potential employee; after hiring thenew potential employee, collecting employment performance informationfor the new potential employee; adding the employment performanceinformation for the new potential employee to the database; andmodifying the artificial intelligence-based model based on the pre-hireand employment performance information for the new potential employee.54. A method for providing an automated hiring recommendation servicefor an employer, the method comprising: stationing a plurality ofelectronic devices at a plurality of employer sites, wherein theelectronic devices are operable to accept directly from one or more jobapplicants answers to questions presented at the electronic devices;sending the answers of at least one of the job applicants to a remotesite for analysis; applying an artificial intelligence-based predictivemodel to the answers of the least one of the job applicant to generatean automated hiring recommendation; and automatically sending the hiringrecommendation to the employer.